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RELATED APPEALS AND INTERFERENCES 
The Board's decision in the present Appeal will not directly affect, or be directly affected, 
or have any bearing on any other appeals or interferences known to the appellant, or to the 
Applicant's legal representative. 

STATUS OF CLAIMS 
Claims pending in the application: 1 1-15 and 27-30. 
Claims allowed: none 
Claims canceled: 1-10, 16-26 and 31 
Claims rejected: 11-15 and 27-30. 
Claims appealed: 11-15 and 27-30. 

STATUS OF AMENDMENTS 
An Amendment After Final Rejection was filed on February 26, 2007 and a Supplemental 
Amendment After Final Rejection was filed on March 27, 2007. Both amendments were entered 
by the Examiner. 

SUMMARY OF CLAIMED SUBJECT MATTER 
Claim 1 1 is drawn to a plurality of musical apparatus communicating with each other. 
Figure 4 illustrates an exemplary embodiment where, for example, a plurality of musical devices 
10' communicates with each other. At least two of the plurality of musical devices includes an 
audio score synthesis mechanism, an audio score mixing mechanism , and an audio score input 
mechanism to create a playable audio score (e.g., specification, page 6, line 31 - page 7, line 16; 
and page 13, lines 1-4). 

Claim 27 is drawn to a plurality of portable musical apparatus (e.g., 10' of Figure 4) 
coupled through a wireless network, where each of the musical apparatus includes an audio score 
synthesis mechanism, an audio playing mechanism, an audio input mechanism, and an audio 
score mixing mechanism (e.g., specification, page 6, line 31-page 7, line 16; and page 13, lines 1- 
4). The mixing mechanism may mix a first audio score from the synthesis mechanism with a 
second audio score from the input mechanism to produce a playable audio score. 



APPELLANT'S BRIEF 



Page 2 



SERIAL NO. 10/684,167 



GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 
Whether claims 11-15 and 27-30 are unpatentable under 35 U.S.C. 103(a) as being 
obvious over Ito (U.S. Patent Publication No. 2003/0121401) in view of Sitrick (U.S. Patent No. 
6,084,168). 

ARGUMENT 
Claim Rejections - 35 USC §103 
Claims 1 1-15 and 27-30 are rejected under 35 U.S.C. §103(a) as being obvious 
over Ito in view of Sitrick. Applicants traverse the rejection. 

Claims 27 and 1 1 

claim 27 recites plural portable musical apparatus in physically separated proximity with 
each other and capable of two-way communication therebetween of an audio score over said 
wireless network, each musical apparatus including: . . . an audio score mixing m echanism. 
Claim 1 1 includes similar limitations. 

The Ito reference discloses a system in which only one apparatus on the network has a 
mixer. The Examiner acknowledges this in the final office action, but alleges that it would have 
been obvious to adapt the Ito/Sitrick combination so that each device would have a mixing 
mechanism, as it has been held that mere duplication of working parts does not constitute 
nonobviousness and refers to MPEP 2144.04. The Examiner has essentially extracted a "per se" 
rule of obviousness from In re Harza, 214 F.2d 669, 124 USPQ 378 (CCPA 1960), which is 
referred to in the relevant section of MPEP 2144.04. However, The Court Of Appeals For The 
Federal Circuit has held that the precedents do not establish any perse rules of obviousness, and 
that reliance on per se rules of obviousness is legally incorrect and must cease. In re Ochiai, 37 
USPQ2d 1127, 1133 (CAFC 1995). Instead, section 103 requires a fact-intensive comparison of 
the claimed invention with the teachings of the prior art, as done below. 

Ito Figure 1 discloses a mixer apparatus 40, which includes a mixer and is wirelessly 
coupled to a plurality of musical apparatus 10 to 30. The plurality of musical apparatus 10 to 30 
lacks a mixing mechanism. Audio signals and MIDI data from music apparatus 10 to 30 are 
transmitted to mixer apparatus 40, where those are mixed to produce new music tones. Put 
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differently, Ito's mixer apparatus 40 mixes audio signals and MIDI data received from the 
plurality of musical apparatus 10, 20, and 30 to produce new music tones. 

In contrast, claim 27 recites a plurality of portable musical apparatus, where each 
apparatus including an audio score mixing mechanism. For example, each of the musical devices 
10' of Figure 4 may include an audio score mixing mechanism. Including a mixing mechanism 
in each of the musical apparatus makes possible novel and unobvious features having unexpected 
advantages over the prior art. Such a feature, for example, makes possible a real-time peer-to- 
peer musical jam session, a real-time peer-to-peer 'swarm,' or an ad-hoc musical jam session 
(specification, page 6, lines 19-20 and page 7, lines 10-12). Ito's system, in which only one 
musical apparatus includes a mixer, can not perform any such musical jam sessions. Nor does 
Ito provide any reason for creating an apparatus to accommodate such musical jam session. 

Additionally, Ito's system needs only one mixer (mixer 40) to work effectively. Ito does 
not provide any reason to include a mixer in each of the plurality of musical apparatus 10 to 30, 
as only one mixer 40 is sufficient for Ito's system. The Supreme Court has recently held that 
while determining obviousness for a §103 rejection, it is "important to identify a reason that 
would have prompted a person of ordinary skill in the relevant field to combine the elements" in 
the prior art to achieve the claimed invention and such a reasoning should be made explicit. KSR 
Int 7 Co. v. Teleflex, Inc., No 04-1350 (U.S. Apr. 30, 2007). The Examiner has not identified any 
motivation, suggestion, or reason in Ito or any other references to include a mixing mechanism in 
each of the plurality of musical apparatus. The rejection, thus, is based on an impermissible 
hindsight reconstruction using the Applicant's disclosure as a roadmap to achieve the claimed 
invention. Hence, a prima facie case of obviousness has not been established. 

For at least these reasons, claims 1 1 and 27 should be allowed along with their associated 
dependent claims. 
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CONCLUSION 

Applicant requests that the rejection of claims 1 1-15 and 27-30 be reversed. 
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CLAIMS APPENDIX 



The following claims are involved in the appeal. 

1 1 . (Rejected) A system of music devices operatively coupled together, the system 
comprising: 

plural apparatus in physical proximity with each other and capable of at least one-way 
communication therebetween of an audio score, 

at least two such apparatus comprising: 
an audio score synthesis mechanism including a playing mechanism for playing the synthesized 
audio score; 

an audio score mixing mechanism coupled with said synthesis mechanism for mixing 
plural audio scores to produce another audio score having components of each of the plural audio 
scores; and 

an audio score input mechanism coupled with said mixing mechanism to provide one or 
more input audio scores thereto for mixing with the synthesized and outplayed played audio 
score, 

said synthesis mechanism, said mixing mechanism and said input mechanism being operable in 
real time to create a playable audio score having components of plural audio scores produced by 
said plural proximate apparatus. 

1 2 . (Rej ected) The system of claim 1 1 , wherein the audio score for transmitting by 
said transmit mechanism is in the form of digital data. 

13. (Rejected) The system of claim 12, wherein the digital data is formatted in 
accordance with a musical instrument digital interface (MIDI) standard. 

14. (Rejected) The system of claim 11, wherein said audio score synthesis mechanism 
of said at least two such apparatus further includes a playing mechanism for playing the 
synthesized audio score and further comprising at least another such apparatus operatively 
coupled with said at least two such apparatus, said at least another such apparatus comprising: 



APPELLANT'S BRIEF 



Page 6 



SERIAL NO. 10/684,167 



an audio score output playing mechanism; and 

a transmit mechanism for wirelessly transmitting an audio score to said at least two such 
apparatus for mixing thereby; 

wherein said at least another apparatus further comprises: 

a second audio score synthesis mechanism operatively coupled with said output 
mechanism for synthesizing an audio score for transmitting by said transmit mechanism; 

a second audio score mixing mechanism coupled with said second synthesis 
mechanism for mixing plural audio scores to produce another audio score having components of 
each of the plural audio scores; and 

a second audio score input mechanism coupled with said second mixing 
mechanism to provide one or more input audio scores thereto for mixing with the synthesized 
and played audio score, 

said second synthesis mechanism, said second mixing mechanism and said second input 
mechanism being operable in real time to create a playable audio score having components of 
plural audio scores produced by said plural proximate apparatus. 

1 5 . (Rej ected) The network of claim 1 4 wherein at least one of said plural apparatus 
further comprises a controller configurable as a master controller and at least another of said 
plural apparatus further comprises a controller configurable as a slave controller wherein said 
master controller is capable of dictating a mode of operation of said network to said slave 
controller. 

27. (Rejected) A musical system comprising: 
a wireless network; and 

plural portable musical apparatus in physically separated proximity with each other and 
capable of two-way communication therebetween of an audio score over said wireless network, 
each musical apparatus including: 

an audio score synthesis mechanism; 

an audio playing mechanism coupled with said network; 

an audio input mechanism coupled with said network; and 
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an audio score mixing mechanism coupled with said synthesis mechanism, said 
input mechanism and said playing mechanism, said mixing mechanism configured to mix a first 
audio score from said synthesis mechanism with a second audio score from said input 
mechanism to produce in real time a playable audio score having components of each of the first 
and second audio scores. 

28. (Rejected) The musical system of claim 27, wherein the playable audio score is in 
the form of digital data. 

29. (Rejected) The musical system of claim 28, wherein the digital data is formatted 
in accordance with a musical instrument digital interface (MIDI) standard. 

30. (Rejected) The musical system of claim 29, wherein said wireless network takes 
the form of a WiFi or Bluetooth network. 
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EVIDENCE APPENDIX 



Copies of the following references are attached: 

U.S. Patent Publication No. 2003/0121401 ("Ito") 
U.S. Patent No. 6,084,168 ("Sitrick") 
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RELATED PROCEEDINGS APPENDIX 



None. 
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(57) ABSTRACT 
A plurality of music apparatus 10 to 30 such as an electronic 
musical instrument and a microphone apparatus arc con- 
nected by wireless to a mixer apparatus 40. A Bluetooth 
module is adopted as wireless communication means to 
construct a piconet with mixer apparatus 40 functioning as 
a master and music apparatus 10 to 30 functioning as slaves. 
Audio signals and MIDI data from music apparatus 10 to 30 
are transmitted by wireless to mixer apparatus 40 through 
isosynchronous communication procedure using Bluetooth 
modules 11, 21, 31, 41. In mixer apparatus 40, with regard 
to the MIDI data, music tone signals based on the MIDI data 
are produced, whereafter the produced music tone signals 
and the aforesaid audio signals transmitted by wireless are 
mixed. Wiring by means of cables between a plurality of 
music apparatus and a mixer apparatus is abolished, thereby 
eliminating the cumbersomeness of wiring and the restric- 
tions accompanying the wiring. 
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MIXER APPARATUS AND MUSIC APPARATUS 
CAPABLE OF COMMUNICATING WITH THE 
MIXER APPARATUS 

BACKGROUND OF THE INVENTION 
[0001] i. Field of the Invention 

[0002] The present invention relates to a mixer apparatus 
for inputting audio signals or audio signal producing signals 
respectively produced in a plurality of music apparatus and 
for mixing the input audio signals or audio signals produced 
on the basis of the input audio signal producing signals, as 
well as a music apparatus capable of wireless communica- 
tion with the mixer apparatus. 

[0003] 2. Description of the Background Art 

[0004] Hitherto, mixer apparatus for mixing audio signals 
from a plurality of music apparatus such as an electronic 
musical instrument and a microphone apparatus for output 
are well known. 

[0005] However, the aforementioned conventional mixer 
apparatus are connected to the plurality of music apparatus 
by means of cables, giving rise to problems such as cum- 
bersome wiring and connection of the cables and the restric- 
tions imposed by the cables on the placement of the music 
apparatus and the mixer apparatus. 

SUMMARY OF THE INVENTION 

[0006] The present invention has been made in order to 
cope with the aforementioned problems of the prior art, and 
an object thereof is to provide a mixer apparatus for input- 
ting audio signals or audio signal producing signals from a 
plurality of music apparatus by wireless without the use of 
cables and for mixing the input audio signals or audio 
signals produced on the basis of the input audio signal 
producing signals. Another object of the present invention is 
to provide a music apparatus capable of wireless commu- 
nication with a mixer apparatus such as mentioned above 
and a computer readable program applied to the mixer 
apparatus. 

[0007] In order to achieve the aforementioned objects, a 
characteristic feature of the present invention lies in a mixer 
apparatus for inputting audio signals or audio signal pro- 
ducing signals respectively produced in a plurality of music 
apparatus and for mixing the input audio signals or audio 
signals produced on the basis of the input audio signal 
producing signals, said mixer apparatus comprising a wire- 
less communication section capable of wireless communi- 
cation with the plurality of music apparatus by allowing the 
plurality of music apparatus to function as slaves and 
allowing the mixer apparatus itself to function as a master, 
said wireless communication section respectively receiving 
the audio signals or audio signal producing signals that are 
transmitted from the plurality of music apparatus; and a 
mixing section for mixing the audio signals received by the 
wireless communication section or the audio signals pro- 
duced on the basis of the audio signal producing signals 
received by the wireless communication section. 

[0008] In this case, the wireless communication section 
respectively issues requests to the plurality of music appa- 
ratus for transmittance of the audio signals or audio signal 
producing signals, and respectively receives the audio sig- 



nals or audio signal producing signals that are transmitted 
from the plurality of music apparatus in response to the 
requests for transmittance Further, as means for wireless 
communication between the plurality of music apparatus 
and the mixer apparatus, one can use, for example, a 
wireless communication device according to the Bluetooth 
(registered trademark) standard. Further, when audio signal 
producing signals are transmitted from the music apparatus 
to the mixer apparatus, audio signals may be produced in an 
audio signal producing section comprised within the mixing 
section on the basis of the audio signal producing signals, 
and the produced audio signals may be mixed. 

[0009] According to this feature, the audio signals or the 
audio signal producing signals from the plurality of music 
apparatus are supplied to the mixer apparatus by wireless, 
thereby eliminating the need for connecting the plurality of 
music apparatus to the mixer apparatus by means of cables. 
This saves the labor of wiring and connection of the cables, 
and the placement of the music apparatus and the mixer 
apparatus can be made freely without being restricted by the 
cables. Further, since the mixer apparatus inputs the audio 
signals or the audio signal producing signals from a plurality 
of music apparatus, traffic (transfer of information) can be 
controlled efficiently by allowing the mixer apparatus to 
function as a master and allowing the plurality of music 
apparatus to function as slaves. 

[0010] Further, another characteristic feature of the 
present invention lies in that the wireless communication 
section receives the audio signals or audio signal producing 
signals from the plurality of music apparatus by isochronous 
communication procedure. In this case, isochronous com- 
munication (isochronous transfer) procedure makes use of 
an ACL link (asynchronous connection-less link). This fea- 
ture allows that, if the number of music apparatus is small, 
the audio signals or the audio signal producing signals can 
be sent at a comparatively high transfer rate, so that the 
communication can be made with comparatively less delays. 

[0011] Further, another characteristic feature of the 
present invention lies in that the mixer apparatus further 
comprises mixed signal transmitting section for transmitting 
the audio signals mixed in the mixing section to the plurality 
of music apparatus via the aforesaid wireless communica- 
tion section. According to this feature, the results of mixing 
the plurality of audio signals are sent to each music appa- 
ratus by wireless, so that the aforesaid results of mixing can 
be monitored at the position of each music apparatus. 

[0012] Further, another characteristic feature of the 
present invention lies in that the aforesaid wireless commu- 
nication section transmits the audio signals mixed in the 
mixing section to the plurality of music apparatus by broad- 
cast communication procedure (multiple address communi- 
cation procedure). According to this feature, the results of 
mixing a plurality of audio signals are transmitted by 
broadcast communication, so that the traffic can be con- 
trolled efficiently without increasing the traffic amount. 

[0013] Further, another characteristic feature of the 
present invention lies in that the mixer apparatus further 
comprises communication condition setting section for set- 
ting conditions of communication with the plurality of music 
apparatus in a state in which a wireless connection is 
established between the mixer apparatus and the plurality of 
music apparatus. In this case, the communication conditions 
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are, for example, selection of the type of music apparatus 
from which the audio signals or audio signal producing 
signals are to be input into the mixer apparatus, selection of 
the type of signals (audio signals or audio signal producing 
signals) which arc to be supplied from the music apparatus 
to the mixer apparatus, and selection of the music apparatus 
to which the results of mixing the plurality of audio signals 
are to be output. This feature allows that, even if the 
combination of a plurality of music apparatus supplied to the 
mixer apparatus is changed, one can meet the change 
speedily. 

[0014] Further, another characteristic feature of the 
present invention lies in that the mixer apparatus comprises 
wired input section connected by wire to a different music 
apparatus other than the plurality of music apparatus, for 
wired input of audio signals or audio signal producing 
signals for producing audio signals that are output from the 
different music apparatus, wherein the aforesaid mixing 
section also mixes the audio signals input by the wired input 
section or the audio signals produced on the basis of the 
audio signal producing signals input by the wired input 
section, in addition to the audio signals received by the 
wireless communication section or the audio signals pro- 
duced on the basis of the audio signal producing signals 
received by the wireless communication section. 

[0015] This feature allows that, even if a music apparatus 
incapable of wireless communication with the mixer appa- 
ratus is present, the music apparatus can be connected by 
wire to the mixer apparatus, whereby audio signals from this 
music apparatus connected by wire or the audio signals 
produced on the basis of the audio signal producing signals 
from this music apparatus can be mixed as well by the mixer 
apparatus. As a result of this, this mixer apparatus can be 
applied to a variety of music apparatus. 

[0016] Further, another characteristic feature of the 
present invention lies in that the mixer apparatus further 
comprises audio signal generating section for generating 
audio signal independently from the aforesaid plurality of 
music apparatus, wherein the aforesaid mixing section also 
mixes the audio signals generated by the audio signal 
generating section, in addition to the audio signals received 
by the wireless communication section or the audio signals 
produced on the basis of the audio signal producing signals 
received by the wireless communication section. According 
to this feature, more audio signals can be mixed, whereby a 
more opulent music can be realized. 

[0017] Further, another characteristic feature of the 
present invention lies in a music apparatus capable of 
wireless communication with a mixer apparatus that mixes 
a plurality of audio signals, wherein the music apparatus 
comprises mixing signal generating section for generating 
the audio signals that will be subjected to mixing or audio 
signal producing signals for producing the audio signals that 
will be subjected to mixing; a wireless communication 
section for transmitting by wireless to the mixer apparatus 
the audio signals or the audio signal producing signals 
generated by the mixing signal generating section and for 
receiving mixed signals mixed by the mixer apparatus and 
transmitted by wireless from the mixer apparatus, said 
mixed signals including the audio signals transmitted by 
wireless from the music apparatus or the audio signals 
produced on the basis of the audio signal producing signals 



transmitted by wireless from the music apparatus; and 
reproduction section for reproducing the audio signals 
received by the wireless communication section. 

[0018] In this case as well, as means for wireless commu- 
nication between the music apparatus and the mixer appa- 
ratus, one can use, for example, a wireless communication 
device according to the Bluetooth standard. Further, when 
audio signal producing signals are transmitted from the 
music apparatus to the mixer apparatus, audio signals may 
be produced in an audio signal producing section comprised 
within the mixing section on the basis of the audio signal 
producing signals, and the produced audio signals may be 
mixed. 

[0019] This feature as well eliminates the need for con- 
necting the music apparatus to the mixer apparatus by means 
of cables, and saves the labor of wiring and connection of the 
cables. Also, the placement of the music apparatus and the 
mixer apparatus can be made freely without being restricted 
by the cables. Furthermore, since the music apparatus inputs 
and reproduces the results of mixing the plurality of audio 
signals in the mixer apparatus, the aforesaid results of 
mixing can be monitored at the position of the music 
apparatus. 

[0020] Further, another characteristic feature of the 
present invention lies in a computer readable program that is 
applied to a mixing apparatus and music apparatus for 
allowing the mixing apparatus and music apparatus to 
perform the aforementioned functions. According to this 
feature, the aforementioned various functions can be imple- 
mented easily by the mixing apparatus and music apparatus 
having a wireless communication function. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0021] FIG. 1 is a block diagram illustrating a network 
according to one embodiment of the present invention; 

[0022] FIG. 2 is a functional block diagram illustrating 
the network of FIG. 1 in further detail; 

[0023] FIG. 3 is a block diagram illustrating an embodi- 
ment of a music apparatus (electronic musical instrument) 
and a mixer apparatus of FIGS. 1 and 2; 

[0024] FIG. 4 is a flowchart showing the former part of a 
program executed by the mixer apparatus and the music 
apparatus of FIGS. 1 and 2 and related to link setting and 
data transmission/reception; and 

[0025] FIG. 5 is a flowchart showing the latter part of the 
program. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0026] Hereafter, one embodiment of the present inven- 
tion will be described with reference to the attached draw- 
ings. FIG. 1 is a block diagram illustrating a network 
according to this embodiment. 

[0027] This network is constituted with a plurality of 
music apparatus 10 to 30 and a mixer apparatus 40 respec- 
tively capable of wireless comm unication with these music 
a pparatus lOlo 30. Music apparatus 10 to 30 produce audio 
signals such as music tone signals or produce audio signal 
producing signals (for example, MIDI data) such as key-on 
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signals, key-off signals, tone color control signals, and tone 
volume control signals that are used for production of these 
audio signals. Mixer apparatus 40 inputs audio signals or 
audio signal producing signals from plural music apparatus 

10 to 30, and mixes the audio signals or audio signals 
produced on the basis of the audio signal producing signals 
for output. 

[0028] These music apparatus 10 to 30 and mixer appa- 
ratus 40 respectively include, as a wireless communication 
section, Bluetooth (registered trademark) modules 11, 21, 
31, 41 that allows wireless communication with each other 
in accordance with the Bluetooth communication standard. 
The wireless communication according to the Bluetooth 
communication standard provides data exchange between 
plural apparatus with the use of a spectrum diffusion pro- 
cedure of frequency hopping type. Further, in the wireless 
communication using this Bluetooth communication stan- 
dard, a wireless network called "piconet" is constructed 
which is made of one master and one or more slaves, where 
Bluetooth modules belonging to one and the same piconet 
are in a synchronized state with each other in the frequency 
axis and in the time axis. 

[0029] Further, in the Bluetooth communication standard, 
one of two types of communication links, which are an SCO 
(synchronous connection-oriented) link and an ACL (asyn- 
chronous connection-less) link, is selected for use in accor- 
dance with a setting. Furthermore, the communication in this 
ACL link is set to use one procedure selected from the 
asynchronous communication procedure, the isochronous 
communication (isochronous transfer) procedure, and the 
broadcast communication procedure (multiple address com- 
munication procedure). 

[0030] Here, among the above-described characteristics of 
the Bluetooth communication standard, this embodiment is 
characterized by adopting a piconet construction including 
one master and plural slaves as well as the isochronous 
communication procedure and the broadcast communication 
procedure in the ACL link. Various wireless communication 
techniques conforming to a communication standard having 
the aforesaid characteristics can be applied to the present 
invention even if the techniques do not conform to the 
Bluetooth communication standard. 

[0031] The aforesaid network of FIG. 1 will be further 
detailed using the functional block diagram of FIG. 2 by 
raising specific examples of music apparatus 10 to 30. Here, 
the illustrated arrows drawn in solid lines denote audio 
signals and the illustrated arrows drawn in broken lines 
denote MIDI data. 

[0032] Music apparatus 10 is constituted with an elec- 
tronic musical instrument and produces MIDI data for 
output. This music apparatus 10 is provided with a MIDI 
data generator 12 that generates MIDI data, and the MIDI 
data generated by MIDI data generator 12 are transmitted by 
wireless to mixer apparatus 40 by Bluetooth module 11. On 
the other hand, the audio signals transmitted by wireless 
from mixer apparatus 40 are received by Bluetooth module 

11 and supplied to sound system 15 via decoder 13 and D/A 
converter 14. Decoder 13 decodes (decompresses) the audio 
signals that are encoded (compressed) by mixer apparatus 40 
and outputs the decoded audio signals. Further, music appa- 
ratus 10 includes a microcomputer 16, and microcomputer 
16 performs various functions in music apparatus 10 by a 
program process. 



[0033] Music apparatus 20 also is constituted with an 
electronic musical instrument and produces and outputs 
digital music tone signals (audio signals). This music appa- 
ratus 20 is provided with a MIDI data generator 22 that 
generates MIDI data and a tone generator circuit 23 that 
produces and outputs digital music tone signals (audio 
signals) on the basis of the aforesaid generated MIDI data. 
These digital music tone signals are encoded (compressed) 
by encoder 24 and transmitted by wireless to mixer appa- 
ratus 40 by Bluetooth module 21. On the other hand, the 
audio signals transmitted by wireless from mixer apparatus 
40 are received by Bluetooth module 21 and supplied to 
sound system 27 via decoder 25 and D/A converter 26. 
Decoder 25 decodes (decompresses) and outputs the audio 
signals that are encoded (compressed) by mixer apparatus 40 
as well. Further, in this case as well, music apparatus 20 
includes a microcomputer 28, and microcomputer 28 per- 
forms various functions in music apparatus 20 by a program 
process. 

[0034] Music apparatus 30 is constituted with a micro- 
phone apparatus and is provided with a microphone 32 that 
converts acoustic signals such as human voices and tones of 
musical instruments into audio signals by acoustic/electric 
conversion for output. These audit) signals converted by 
microphone 32 are converted into digital audio signals by 
A/D converter 33. These converted digital audio signals are 
encoded (compressed) by encoder 34 and transmitted by 
wireless to mixer apparatus 40 by Bluetooth module 31. On 
the other hand, the audio signals transmitted by wireless 
from mixer apparatus 40 are received by Bluetooth module 
31 and supplied to sound system 37 via decoder 35 and D/A 
converter 36. Decoder 35 decodes (decompresses) and out- 
puts the audio signals that are encoded (compressed) by 
mixer apparatus 40 as well. Further, in this case as well, 
music apparatus 30 includes a microcomputer 38, and 
microcomputer 38 performs various functions in music 
apparatus 30 by a program process. 

[0035] Mixer apparatus 40 is provided with a Bluetooth 
module 41 that receives the MIDI data, digital music tone 
signals, and digital audio signals respectively transmitted by 
wireless from music apparatus 10 to 30. These received 
MIDI data, digital music tone signals, and digital audio 
signals are respectively output to tone generator circuit 42a, 
decoder 43a, and decoder 436, respectively. Tone generator 
circuit 42a produces and outputs digital music tone signals 
(one type of audio signals) on the basis of the MIDI data. 
Decoders 43a, 43f> decode (decompress) and output the 
digital music tone signals and digital audio signals respec- 
tively encoded (compressed) by music apparatus 20, 30. 

[0036] Characteristics control circuits 44a to 44c are 
respectively connected to tone generator circuit 42a and 
decoders 43a, 43f>. Characteristics control circuits 44a to 
44c respectively perform a compressing process, a limiting 
process, an equalizing process, and the like on the supplied 
digital music tone signals and digital audio signals for 
output. The compressing process is a process of changing 
the dynamic range of the input signals. The limiting process 
is a process of restraining the maximum level of the input 
signals. Hie equalizing process is a process of changing the 
frequency characteristics of the input signals. 

[0037] Level setting circuits 45a to 45c are connected to 
respective outputs of characteristics control circuits 44a to 
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44c. Level setting circuits 45a to 45c change the input signal 
levels in various ways for output. The outputs of level setting 
circuits 45a to 45c are input into additive synthesis circuits 
46a to 46c. Additive synthesis circuits 46a to 46c are each 
provided with a gate circuit that selectively outputs the 
signals from level setting circuits 45a to 45c, and the results 
of addition from the additive synthesis circuit of the previ- 
ous stage (additive synthesis circuit located on the illustrated 
left side) are added to the signals selectively output from the 
aforesaid gate circuit and output to the additive synthesis 
circuit of the following stage (additive synthesis circuit 
located on the illustrated right side). 

[0038] Further, mixer apparatus 40 is provided with a 
MIDI data generator 47 that outputs MIDI data indepen- 
dently with no relation to the outside music apparatus 10 to 
30 and a tone generator circuit 426 that produces and outputs 
digital music tone signals (one type of audio signals) on the 
basis of the aforesaid generated MIDI data. The digital 
music tone signals output from tone generator circuit 426 are 
output to additive synthesis circuit 46d via characteristics 
control circuit 44d and level setting circuit 45d that are 
constructed in the same manner as the aforesaid character- 
istics control circuits 44a to 44c and level setting circuits 
45a to 45c. 

[0039] The output from additive synthesis circuit 46d of 
the final stage is input into level setting circuit 51. Level 
setting circuit 51 changes the input signal levels in various 
ways for output. The output of level setting circuit 51 is 
connected to sound system 53 via D/A converter 52 that 
converts digital signals to analog signals. 

[0040] The respective outputs of level setting circuits 45a 
to 45d arc also connected to additive synthesis circuits 54a 
to 54d that arc constructed in the same manner as the 
aforesaid additive synthesis circuits 46a to 46d. Here, in 
additive synthesis circuits 54a to 54d, the additive synthesis 
circuit of the previous stage corresponds to the one located 
on the illustrated right side, and the additive synthesis circuit 
of the following stage corresponds to the one located on the 
illustrated left side. The output from additive synthesis 
circuit 54a of the final stage is encoded (compressed) by 
encoder 55 and respectively output to music apparatus 10 to 
30 via Bluetooth module 41. Furthermore, mixer apparatus 
40 includes a microcomputer 56, and microcomputer 56 
performs various functions in mixer apparatus 40 by a 
program process. 

[0041] Next, one embodiment of the electronic musical 
instruments used as the aforesaid music apparatus 10, 20 and 
a mixer apparatus of electronic musical instrument function 
incorporating type used as mixer apparatus 40 will be 
described with reference to FIG. 3. 

[0042] The apparatus of this type is provided with a 
keyboard 61 made of a plurality of keys, a panel operator 
group 62 disposed on an operation panel, and a display 63. 
Each key indicates the generation of a music tone signal, and 
the pressing/depressing of each key is detected by a detec- 
tion circuit 64 connected to bus 60. Panel switch group 62 
is operated mainly in relation to the display on display 63, 
and selects or controls various functions in this apparatus, 
such as the music tone elements (pitch shift, tone color, tone 
volume, and the like) of the generated music tone signals, 
the effects imparted to the music tone signals, the state of 
mixing a plurality of music tone signals, the generation of 



automatic accompaniment tones, and the reproduction of 
automatic play tones. These operations of panel operator 
group 62 are detected by a detection circuit 65 connected to 
bus 60. Display 63 displays symbols, characters, and the like 
for selecting and setting various functions in this apparatus 
under control of a display circuit 66 connected to bus 60. 

[0043] Also, a CPU 71, a timer 72, a ROM 73, a RAM 74, 
and an external storage device 75 are connected to bus 60. 
CPU 71 executes various programs including the programs 
shown in FIGS. 4 and 5 stored in ROM 73, RAM 74, or 
external storage device 75 in collaboration with timer 72 and 
RAM 74, thereby realizing various functions of this appa- 
ratus. External storage device 34 includes recording media 
having a comparatively large capacity such as a hard disk 
HD, a flexible disk FD, a compact disk CD, a magneto- 
optical disk MO, a digital versatile disk DVD, and a semi- 
conductor memory, as well as a drive unit for each of the 
recording media. These recording media store various pro- 
grams as well as various data used for implementing various 
functions of this apparatus, such as, various control data for 
producing music tone signals and for controlling the pro- 
duced music tone signals, and control data for controlling 
the generation of music tone signals (automatic performance 
data made of MIDI data). 

[0044] Also, a MIDI interface circuit 76 and a Bluetooth 
module 77 are connected to bus 60. MIDI interface circuit 
76 inputs MIDI data from other music apparatus 78 such as 
electronic musical instruments and sequencers connected by 
wire, and outputs MIDI data to the aforesaid other music 
apparatus 78. Bluetooth module 77 receives audio signals 
and MIDI data from Bluetooth modules 79 incorporated in 
other music apparatus such as electronic musical instru- 
ments, sequencers, and microphone apparatus connected by 
wireless, and transmits audio signals and MIDI data to 
Bluetooth modules 79 incorporated in the aforesaid other 
music apparatus. 

[0045] Also, a tone generator circuit 81 and a mixing 
circuit 82 are connected to bus 60. Tone generator circuit 81 
produces music tone signals in accordance with the control 
signals (MIDI data) input via bus 60 and representing 
key-on, key-off, and others for output to mixing circuit 82. 
In this case, the aforesaid control signals (MIDI data) are 
supplied by performance operatioas on keyboard 61 and 
reproduction of music data stored in external storage device 
75 by automatic play. Further, MIDI data supplied from 
other MIDI apparatus 78 to MIDI interface circuit 76 by 
wire and MIDI data supplied from other Bluetooth modules 
79 to Bluetooth module 77 by wireless are supplied to tone 
generator circuit 81 via bus 60. 

[0046] Mixing circuit 82 inputs digital music tone signals 
of plural series supplied from tone generator circuit 81 
through channels that are different series by series, and 
mixes the plural music tone signals after controlling the 
characteristics and levels of the music tone signals for each 
channel. Also, an audio input circuit 83 connected by wire 
to other music apparatus 84 is connected to mixing circuit 
82. Audio input circuit 83 inputs audio signals from other 
music apparatus (electronic musical instruments, automatic 
play apparatus, microphone apparatus, and the like) by wire 
and outputs the audio signals to mixing circuit 82. Also, 
audio signals transmitted by wireless from other Bluetooth 
modules 79 and received by Bluetooth module 77 are input 
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into mixing circuit 82 via bus 60. Mixing circuit 82 respec- 
tively inputs the audio signals from audio input circuit 83 
and Bluetooth module 77 as well through channels that are 
different from those of the aforesaid music tone signals, 
controls the characteristics and levels of the audio signals at 
each channel, and mixes the audio signals with the aforesaid 
digital music tone signals from tone generator circuit 81. 

[0047] The output of mixing circuit 82 is connected to D/A 
converter 85. D/A converter 85 converts the digital audio 
signals from the mixing circuit into analog audio signals for 
output to sound system 86. Sound system 86 is composed of 
amplifiers 86a, 866, speaker 86c, and headphone 86d. 

[0048] Here, the relationship of music apparatus 10, 20 
and mixer apparatus 40 in FIG. 2 to the afore said music 
apparatus constructed as shown in FIG. 3 will be described. 
First, the relationship between music apparatus 10 in FIG. 
2 and the music apparatus in FIG. 3 will be described. MIDI 
data generator 12 in FIG. 2 corresponds to a device for 
outputting the performance data produced by playing on 
keyboard 61 and a device for reproducing the performance 
data in the music data stored in external storage device 75 in 
FIG. 3. In other words, MIDI data generator 12 in FIG. 2 
corresponds to keyboard 61, detection circuit 64, CPU 71, 
external storage device 75, and others in FIG. 3. Bluetooth 
module 11, D/A converter 14, and sound system 15 in FIG. 
2 correspond to Bluetooth module 77, D/A converter 85, and 
sound system 86 in FIG. 3, respectively. Decoder 13 in FIG. 
2 corresponds to a device for decoding the audio signals 
received by Bluetooth module 77 by a program process, 
namely, to CPU 71, RAM 74, and others in FIG. 3. 
Microcomputer 16 in FIG. 2 corresponds to CPU 71, timer 
72, ROM 73, RAM 74, and external storage device 75 in 
FIG. 3. 

[0049] The relationship between music apparatus 20 in 
FIG. 2 and the music apparatus in FIG. 3 will be described. 
MIDI data generator 12 in FIG. 2 corresponds to a device 
for outputting the performance data produced by playing on 
keyboard 61, a device for reproducing the performance data 
in the music data stored in external storage device 75, a 
device for inputting MIDI data from outside, and others in 
FIG. 3, namely, to keyboard 61, detection circuit 64, CPU 
71, external storage device 75, MIDI interface circuit 76, 
Bluetooth module 77, and others in FIG. 3. Tone generator 
circuit 23 in FIG. 2 corresponds to a device for producing 
music tone signals in accordance with performance data, 
MIDI data, or the like, namely, to tone generator circuit 81 
in FIG. 3. Regarding Bluetooth module 21, decoder 25, D/A 
converter 26, sound system 27, and microcomputer 28 in 
FIG. 2, the same applies as in the case of Bluetooth module 
11, decoder 13, D/A converter 14, sound system 15, and 
microcomputer 16 in FIG. 2 described above. 

[0050] The relationship between mixer apparatus 40 in 
FIG. 2 and the music apparatus in FIG. 3 will be described. 
MIDI data generator 47 in FIG. 2 corresponds to a device 
for outputting the performance data produced by playing on 
keyboard 61 and a device for reproducing the performance 
data in the music data stored in external storage device 75 in 
FIG. 3, namely, to keyboard 61, detection circuit 64, CPU 
71, external storage device 75, and others in FIG. 3. Tone 
generator circuits 42a, 426 in FIG. 2 correspond to a device 
for producing music tone signals in accordance with per- 
formance data, MIDI data, or the like, namely, to tone 



generator circuit 81 in FIG. 3. Decoders 43a, 436 in FIG. 
2 correspond to a device for decoding the audio signals 
received by Bluetooth module 77 by a program process, 
namely, to CPU 71, RAM 74, and others in FIG. 3. Encoder 
55 in FIG. 2 corresponds to a device for encoding the audio 
signals to be output to Bluetooth module 77 by a program 
process, namely, to CPU 71, RAM 74, and others in FIG. 3. 

[0051] Characteristics control circuits 44a to 44d, level 
setting circuits 45a to 45d, 51, additive synthesis circuits 
46a to 46d, 54a to 54a" correspond to a device for controlling 
the characteristics of audio signals by a program process, a 
device for controlling the levels of audio signals by a 
program process, and a device for performing additive 
synthesis of audio signals by a program process, namely, to 
panel switch group 62, detection circuit 65, CPU 71, RAM 
74, mixing circuit 82, and others. Bluetooth module 41, D/A 
converter 52, and sound system 53 in FIG. 2 correspond to 
Bluetooth module 77, D/A converter 85, and sound system 
86 in FIG. 3, respectively. Microcomputer 56 in FIG. 2 
corresponds to CPU 71, timer 72, ROM 73, RAM 74, and 
external storage device 75 in FIG. 3. 

[0052] Further, although an embodiment of music appa- 
ratus (microphone apparatus) 30 in FIG. 2 is not illustrated, 
sound system 37 in this music apparatus 30 corresponds to 
sound system 86 such as shown in FIG. 3, and includes a 
speaker and a headphone. Further, microcomputer 38 in 
FIG. 2 is constructed with circuits similar to CPU 71, timer 
72, ROM 73, RAM 74, and external storage device 75 in 
FIG. 3. 

[0053] Next, the operation of music apparatus 10 to 30 and 
mixer apparatus 40 constructed as shown above will be 
described along the flowcharts of FIGS. 4 and 5. In these 
music apparatus 10 to 30 and mixer apparatus 40, Bluetooth 
modules 11, 21, 31, 41 of apparatus 10 to 40 are set in 
advance so that music apparatus 10 to 30 may function as 
slaves and mixer apparatus 40 may function as a master. 
When the power switches of apparatus 10 to 40 are turned 
on in a predetermined area music apparatus 10 to 40 can 
transmit and receive data with each other, an ACL link is 
established among Bluetooth modules 11, 21, 31, 41. Alter- 
natively, when apparatus 10 to 40 are moved into a prede- 
termined area in a state in which the power switches of 
apparatus 10 to 40 are turned on, an ACL link is established 
among Bluetooth modules 11, 21, 31, 41. In this case, the 
power switch of mixer apparatus 40 functioning as a master 
is turned on first, and thereafter the power switches of music 
apparatus 10 to 30 functioning as slaves are turned on (or the 
slaves are moved into an area where communication with 
the master can be made). This is because, if a Bluetooth 
module functioning as a master is not present, the piconet 
connection is not established. Thus, microcomputers 16, 28, 
38, 56 establish the aforesaid ACL link of Bluetooth mod- 
ules 11, 21, 31, 41 by the processes of steps S10, S20, S30, 
S40. 

[0054] Next, conditions for transmitting and receiving 
signals between music apparatus 10 to 30 and mixer appa- 
ratus 40 are set. In this case, a user operates panel switch 
group 62 while looking at display 63 of music apparatus 10 
to 30 and mixer apparatus 40. Hereafter, the aforesaid setting 
of the conditions for transmitting and receiving signals will 
be described by referring to the above-described case of 
FIG. 2 as an example. In mixer apparatus 40, channels in 
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mixing, input sources, and types of input signals are set as 
a condition for receiving signals, as shown in the following 
Table 1, through the process of step S41 performed by 
microcomputer 56. Further, in the step S41, destinations for 
outputting the results of mixing shown in the following 
Table 2 arc set as a condition for transmitting signals. 

TABLE 1 



Ch input sources type of input signals 

1 music apparatus 10 MIDI 
(electronic musical instrument) 

(Bluetooth module 11) 

2 music apparatus 20 audio 
(electronic musical instrument) 

(Bluetooth module 21) 

3 music apparatus 30 audio 
(microphone) 

(Bluetooth module 31) MIDI 

4 mixer apparatus 40 



[0055] 

TABLE 2 



mixing output destinations 

music apparatus 10 (electronic musical instrument) 
(Bluetooth module 11) 

music apparatus 20 (electronic musical instrument) 

(Bluetooth module 21) 

music apparatus 30 (microphone) 

(Bluetooth module 31) 



[0056] In music apparatus 10 to 30, output destinations 
and types of output signals are set, as shown in the following 
Table 3, by the processes of steps SU, S21, S31 performed 
by microcomputers 16, 28, 38 as a condition for transmitting 
signals. Further, in these processes of steps Sll, S21, S31, 
monitor input sources shown in the following Table 4 are set 
as a condition for receiving signals. 



TABLE 3 


output destinations types of output signals 


mi 


xer apparatus 40 MIDI 


(B 


luetooth module 41) 


[0057] 






TABLE 4 


monitor input sources 




mixer apparatus 40 




(Bluetooth module 41) 



[0058] After the aforesaid process of step S41, microcom- 
puter 56 in step S42 sets a condition for communicating data 
in accordance with the number of connected slaves, the 
types of transmitted and received signals (MIDI/audio sig- 
nals), and others, and sets a condition for encoding the audio 
signals to be transmitted and received. Specifically, if the 
number of connected slaves is large, the quality of the audio 
signals at the time of encoding may be reduced (if the quality 



is low, the amount of data per one channel decreases, so that 
simultaneous transmittance/reception can be made through a 
larger number of channels), while if the number of slaves is 
small, the quality at the time of encoding the audio signals 
may be raised (if simultaneous transmission/reception is 
made through a smaller number of channels, the amount of 
data per one channel can be increased, so that the audio 
signals can be transmitted and received with raised quality 
of encoding). Alternatively, if MIDI is included as the 
transmitted and received signals, the quality of the audio 
signals at the time of encoding may be raised (since the 
amount of transmitted/received data is small in MIDI, the 
quality of the audio signals can be raised by allotting the 
reduced amount to the data transmittance/reception of the 
audio signals). In any case, the encoding condition is vari- 
ably set so that the audio data can be transmitted and 
received with the highest possible quality in accordance with 
the number of connected slaves and the types of transmitted 
and received signals. 

[0059] Then, in step S43, the encoding condition is trans- 
mitted to music apparatus 10 to 30 via Bluetooth module 41. 
In music apparatus 10 to 30, the aforesaid transmitted 
encoding condition is incorporated into microcomputers 16, 
28, 38 via Bluetooth modules 11, 21, 31, whereafter the 
decoding operations and encoding operations in decoders 
13, 25, 35, 43a, 436 and encoders 24, 34, 55 will be 
controlled in accordance with the aforesaid encoding con- 
dition. 

[0060] After the setting of various conditions such as 
described above is finished, when MIDI data are generated 
in MIDI data generator 12 through the process of step S12 
performed by microcomputer 16, Bluetooth module 11 
temporarily stores these MIDI data. 

[0061] Further, in music apparatus 20, when MIDI data are 
generated in MIDI data generator 22 through the process of 
step S22 performed by microcomputer 28, digital music tone 
signals are produced in tone generator circuit 23 on the basis 
of the aforesaid MIDI data by the process of step S23. These 
digital music tone signals are encoded in encoder 24 through 
the process of step S24 and supplied to Bluetooth module 
21, which in turn temporarily stores the aforesaid encoded 
digital music tone signals. 

[0062] Further, in music apparatus 30, when audio signals 
such as human voices and tones of musical instruments are 
input into microphone 32, these audio signals are subjected 
to A/D conversion in A/D converter 33. These digital audio 
signals subjected to A/D conversion are then encoded in 
encoder 34 through the process of step S32 performed by 
microcomputer 28 and supplied to Bluetooth module 31, 
which in turn temporarily stores the aforesaid encoded 
digital music tone signals. 

[0063] When a request for data transmittance is issued 
from mixer apparatus 40 to music apparatus 10 through the 
process of step S44 performed by microcomputer 56 in this 
state, music apparatus 10 transmits the aforesaid MIDI data 
temporarily stored in Bluetooth module 11 to mixer appa- 
ratus 40 through the process of step S13 performed by 
microcomputer 16. Mixer apparatus 40 receives these trans- 
mitted MIDI data at Bluetooth module 41. 

[0064] In mixer apparatus 40, the MIDI data received at 
Bluetooth module 41 are sent to tone generator circuit 42a 
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through the process of step S45. Tone generator circuit 42a 
then produces digital music tone signals on the basis of these 
MIDI data. 

[0065] Also, when a request for data transmittance is 
issued from mixer apparatus 40 to music apparatus 20 
through the process of step S46 performed by microcom- 
puter 56, music apparatus 20 transmits the aforesaid 
encoded digital music tone signals temporarily stored in 
Bluetooth module 21 to mixer apparatus 40 through the 
process of step S25 performed by microcomputer 28. Mixer 
apparatus 40 receives these transmitted digital music tone 
signals at Bluetooth module 41. These music tone signals are 
then decoded in decoder 43a through the process of step 
S47. 

[0066] Also, when a request for data transmittance is 
issued from mixer apparatus 40 to music apparatus 30 
through the process of step S48 performed by microcom- 
puter 56, music apparatus 30 transmits the aforesaid 
encoded digital audio signals temporarily stored in Blue- 
tooth module 31 to mixer apparatus 40 through the process 
of step S33 performed by microcomputer 38. Mixer appa- 
ratus 40 receives these transmitted digital audio signals at 
Bluetooth module 41. These digital audio signals are then 
decoded in decoder 436 through the process of step S49. 

[0067] Further, in mixer apparatus 40, when MIDI data are 
generated in MIDI generator 47 through the process of step 
S50 of FIG. 5 performed by microcomputer 56, digital 
music tone signals are produced in tone generator circuit 42b 
on the basis of the aforesaid MIDI data through the process 
ofstepS51. 

[0068] Next, the aforesaid produced and decoded digital 
music tone signals and digital audio signals are supplied 
from tone generator circuits 42a, 42b and decoders 43a, 43b 
to characteristics control circuits 44a to 44d constituting the 
mixing circuit through the process of step S52. Character- 
istics control circuits 44a to 44d independently control the 
characteristics of the digital music tone signals and digital 
audio signals from tone generator circuit 42a, decoders 43a, 
436, and tone generator circuit 42b, respectively, for output 
to level setting circuits 45a to 45a", respectively. Level 
setting circuits 45a to 45a 1 independently control the tone 
volume levels of the digital music tone signals and digital 
audio signals having controlled characteristics, respectively, 
for output to additive synthesis circuits 46a to 46c/, respec- 
tively. 

[0069] Additive synthesis circuits 46a to 46d perform 
additive synthesis of these digital music tone signals and 
digital audio signals, and output the synthesized digital 
audio signal to D/A converter 52 via level setting circuit 51. 
D/A converter 52 in turn converts this digital audio signal 
into analog audio signal and supplies the converted analog 
audio signal to sound system 53. Sound system 53 then 
generates the aforesaid analog audio signal. 

[0070] On the other hand, the aforesaid digital music tone 
signals and digital audio signals from level setting circuits 
45a to 45d are also supplied to additive synthesis circuits 
54a to 54^, respectively, and additive synthesis circuits 54a 
to 54d perform additive synthesis of these digital music tone 
signals and digital audio signals for output. 

[0071] Then, through the process of step S53 performed 
by microcomputer 56, the aforesaid digital audio signal 



obtained by additive synthesis of the digital music tone 
signals and digital audio signals is encoded in encoder 55 
and temporarily stored into Bluetooth module 41. This 
digital audio signal temporarily stored in Bluetooth module 
41 is transmitted from the module 41 to music apparatus 10 
to 30 respectively by broadcast communication procedure 
(multiple address communication procedure) through the 
process of step S54. 

[0072] Music apparatus 10 to 30 receive the aforesaid 
transmitted digital audio signal at Bluetooth modules 11 to 
31, respectively. Then, through the processes of steps S14, 
S26, S34 performed by microcomputers 16, 28, 38, the 
aforesaid received digital audio signal is decoded in decod- 
ers 13, 25, 35, respectively. These decoded digital audio 
signals are converted into analog audio signals in D/A 
converters 14, 26, 36, respectively. These analog audio 
signals are then supplied to sound systems 15, 27, 37 for 
generating tones. 

[0073] After the aforesaid processes of steps S14, S26, 
S34, S54, microcomputers 16, 28, 38, 56 return to steps S12, 
S22, S32, S42, respectively, and repeatedly execute the 
aforesaid processes of steps S12, S22, S32, S42 to steps S14, 
S26, S34, S54, thereby continuously executing the aforesaid 
operation of mixing the audio signals. 

[0074] As will be understood from the above description 
of the operations, according to the above -described embodi- 
ment, the audio signals (including the music tone signals) 
and MIDI data from the plurality of music apparatus 10 to 
30 are supplied to mixer apparatus 40 by wireless, thereby 
eliminating the need for connecting the plurality of music 
apparatus 10 to 30 to mixer apparatus 40 by means of cables. 
This saves the labor of wiring and connection of the cables, 
and the placement (arrangement) of music apparatus 10 to 
30 and mixer apparatus 40 can be made freely without being 
restricted by the cables. 

[0075] Further, since mixer apparatus 40 inputs the audio 
signals and MIDI data from the plurality of music apparatus 
10 to 30, traffic (transfer of information) can be controlled 
efficiently by allowing mixer apparatus 40 to function as a 
master and allowing the plurality of music apparatus 10 to 
30 to function as slaves. Specifically, in piconet connection 
of Bluetooth, transmittance and reception of data arc always 
carried out through communication between a master and 
slaves. For this reason, supposing that data are to be trans- 
mitted from one slave to a different slave, one must once 
transmit the data from the one slave to the master and 
thereafter transmit the data from the master to the different 
slave. Supposing that the one slave is a music apparatus and 
the different slave is mixer apparatus 40, the data transmitted 
from music apparatus 10 to 30 are once received by the 
master and thereafter transmitted from the master to mixer 
apparatus 40. If this is carried out, one piece of data must be 
sent twice, thereby increasing the communication traffic and 
increasing the time delay till the piece of data reaches the 
destination. However, if mixer apparatus 40 is the master, 
data can be transmitted from music apparatus 10 to 30 
functioning as slaves to mixer apparatus 40 by one data 
transmittance process, thereby preventing the increase of 
communication traffic and the increase of time delay. 

[0076] Moreover, since mixer apparatus 40 is constructed 
to receive audio signals and MIDI data from the plurality of 
music apparatus 10 to 30 by isochronous communication 
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procedure, the audio signals and MIDI data can be trans- 
mitted at a comparatively high transfer rate, thereby achiev- 
ing a communication with comparatively smaller delays. 
Specifically, in the piconet connection of Bluetooth, there 
are an SCO link and an ACL link, as described before. The 
SCO link is a communication link with three channels at the 
maximum which is suitable for real-time voice communi- 
cation with a predetermined communication speed (64 kbps) 
ensured. On the other hand, the ACL link is a communica- 
tion link which is originally unsuitable for voice communi- 
cation with varying communication speed depending on data 
traffic and others. At first sight, the SCO link may seem 
suitable for mixer apparatus 40; however, the ACL link can 
have seven channels at the maximum with a high maximum 
communication speed (for example, 432.6 kbps at the maxi- 
mum), and can transmit audio data of high tone quality. 
Moreover, in the ACL link, there are the asynchronous 
communication procedure, the isochronous communication 
procedure, and the broadcast communication procedure, and 
among these, the isochronous communication procedure is a 
procedure with comparatively smaller time delays. There- 
fore, in this embodiment, mixer apparatus 40 having a 
comparatively high competence has been realized by adopt- 
ing the isochronous communication procedure of the ACL 
link with comparatively smaller time delays at this commu- 
nication speed. Here, if a high competence is not desired, 
mixer apparatus 40 with three channels at the maximum may- 
be realized by adopting the SCO link. 

[0077] Further, since music apparatus 10 to 30 receive and 
reproduce the audio signals mixed in mixer apparatus 40, the 
results of mixing a plurality of audio signals can be moni- 
tored at the position of each music apparatus 10 to 30. Since 
the transmittancc of audio signals in this case is carried out 
by the broadcast communication procedure (multiple 
address communication procedure), the traffic can be con- 
trolled efficiently without increasing the amount of traffic. 
Specifically, with the broadcast communication procedure, 
the slave side that has received data need not send a response 
notifying the receipt of data to the master, and moreover, the 
same data can be transmitted to a plurality of slaves at a 
time, thereby enhancing the traffic efficiency. Here, since the 
slaves do not send the response notifying the receipt of data 
to the master, there will be no assurance of data reaching the 
destination with certainty; however, the loss of a small 
amount of data will not be a problem as long as the data are 
used for confirming the results of mixing. In this case, a filter 
for smoothing the data may be used in order to prevent noise 
generation caused by the loss of data. 

[0078] Further, the communication condition such as 
described above between mixer apparatus 40 and music 
apparatus 10 to 30 is set through the processes of steps S10, 
Sll, S20, S21, S30, S31, S40, S41. Therefore, even if the 
combination of mixer apparatus 40 with plural music appa- 
ratus 10 to 30 is changed, one can meet the change speedily. 

[0079] Furthermore, although not specifically described in 
the above description of operations using the functional 
block diagram of FIG. 2, mixer apparatus 40 can receive 
input of audio signals also by wire from another music 
apparatus 84 into audio input circuit 83, as shown in FIG. 
3, and these audio signals can be mixed as well. Further, 
mixer apparatus 40 can receive input of MIDI data also by 
wire from another music apparatus 78, as shown in FIG. 3, 
and the audio signals produced in tone generator circuit 81 



on the basis of these MIDI data can be mixed as well. 
Therefore, audio signals and audio signals based on MIDI 
data from other music apparatus without having wireless 
communication means can be mixed as well in mixer 
apparatus 40, whereby more audio signals can be mixed, and 
a more opulent music can be realized. 
[0080] Here, in the above-described embodiment, three 
music apparatus 10 to 30 are connected to mixer apparatus 
40; however, the number of music apparatus connected to 
mixer apparatus 40 is not limited to three but may be a 
different number. Specifically, if a Bluetooth module is to be 
adopted as wireless communication means as in the above- 
described embodiment, seven music apparatus can be con- 
nected by wireless as slaves to mixer apparatus 40, since the 
current piconet of Bluetooth Ver. 1.0 can have seven slaves 
at the maximum. However, if the number of slaves increases, 
the data transfer rate between mixer apparatus 40 and each 
slave decreases, whereby the tone quality decreases. There- 
fore, it is preferable that about three or four music apparatus 
are connected to mixer apparatus 40. However, if the data 
transfer rate increases owing to a future advancement of 
Bluetooth technology, mixing at a high tone quality can be 
achieved even if the number of music apparatus connected 
to mixer apparatus 40 increases. 

[0081] Further, an electronic musical instrument and a 
microphone apparatus are adopted as music apparatus 10 to 
30; however, any apparatus may be adopted as a music 
apparatus as long as the music apparatus can transmit audio 
signals or audio signal producing signals, and the combina- 
tion thereof can be freely made. 

[0082] Further, in the above-described embodiment, 
description has been made only for the case in which two 
lone generator circuits 42a, 42b and two decoders 43a, 43b 
are used in mixer apparatus 40; however, the number of tone 
generator circuits and the number of decoders can be freely 
set. In addition, the number of MIDI data generators 47 for 
generating MIDI data independently from music apparatus 
10 to 30 may be increased. 

[0083] Further, in the above-described embodiment, mixer 
apparatus 40 having an electronic musical instrument func- 
tion, namely mixer apparatus 40 incorporating tone genera- 
tor circuits 42a, 42b that generate music tone signals, is 
adopted; however, a mixer apparatus that does not include an 
electronic musical instrument function and receives only the 
audio signals for mixing can be adopted as mixer apparatus 
40. 

[0084] Further, when a music apparatus functioning as a 
new slave enters the communication range of the piconet 
while mixer apparatus 40 is receiving MIDI data and audio 
signals from music apparatus 10 to 30 such as an electronic 
musical instrument and a microphone apparatus and mixing 
the audio signals, this new music apparatus may be added 
into the piconet so that the new music apparatus may 
participate in the aforesaid mixing of audio signals. At this 
moment, if the new apparatus is an apparatus functioning as 
one of the slaves previously set in mixer apparatus 40, the 
new apparatus may be added into the piconet, while in the 
other cases, the new music apparatus may not be added into 
the piconet. Further, when one or more music apparatus 
(slaves) have gone out of the communication range of the 
piconet while the audio signals are being mixed, or when the 
power switch of the music apparatus is turned off, the music 
apparatus may be excluded from the piconet. 
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[0085] Further, a buffer for accumulating audio data cor- 
responding to a predetermined period of time may be 
provided (for example, the buffer may be disposed at the 
stage previous to each characteristics control circuit 44) in 
order to absorb the data transmittance/reception time delays 
of each channel so that the data of each channel may be 
output in synchronization. This allows that, even if data 
transmittance time delays are present, sounds are not inter- 
rupted, although time delays may occur to some extent. 

[0086] Further, in the above-described embodiment, elec- 
tronic musical instruments having a keyboard are adopted as 
music apparatus 10, 20; however, electronic musical instru- 
ments having performance operators other than a keyboard, 
for example, electronic musical instruments of string instru- 
ment type, wind instrument type, percussion instrument 
type, and the like can be adopted as well. 

[0087] Furthermore, in carrying out the present invention, 
it is not limited to the above-described embodiments or 
modifications thereof, so that various modifications can be 
made as long as they do not depart from the object of the 
present invention. 

What is claimed is: 

1. A mixer apparatus for inputting audio signals or audio 
signal producing signals respectively produced in a plurality 
of music apparatus and for mixing the input audio signals or 
audio signals produced on the basis of the input audio signal 
producing signals, said mixer apparatus comprising: 

a wireless communication section capable of wireless 
communication with said plurality of music apparatus 
by allowing said plurality of music apparatus to func- 
tion as slaves and allowing said mixer apparatus itself 
to function as a master, said wireless communication 
section respectively receiving said audio signals or 
audio signal producing signals that arc transmitted from 
said plurality of music apparatus; and 

a mixing section for mixing the audio signals received by 
said wireless communication section or the audio sig- 
nals produced on the basis of the audio signal produc- 
ing signals received by said wireless communication 
section. 

2. The mixer apparatus according to claim 1, wherein said 
wireless communication section respectively issues requests 
to said plurality of music apparatus for transmittance of said 
audio signals or audio signal producing signals, and respec- 
tively receives said audio signals or audio signal producing 
signals that arc transmitted from said plurality of music 
apparatus in response to said requests for transmittance. 

3. The mixer apparatus according to claim f , wherein said 
wireless communication section receives said audio signals 
or audio signal producing signals from said plurality of 
music apparatus by isochronous communication procedure. 

4. The mixer apparatus according to claim 1, further 
comprising mixed signal transmitting section for transmit- 
ting the audio signals mixed in said mixing section to said 
plurality of music apparatus via said wireless communica- 
tion section. 

5. The mixer apparatus according to claim 4, wherein said 
wireless communication section transmits the audio signals 
mixed in said mixing section to said plurality of music 
apparatus by broadcast communication procedure. 

6. The mixer apparatus according to claim 1, further 
comprising communication condition setting section for 



setting conditions of communication with said plurality of 
music apparatus in a state in which a wireless connection is 
established between said mixer apparatus and said plurality 
of music apparatus. 

7. The mixer apparatus according to claim f, further 
comprising wired input section connected by wire to a 
different music apparatus other than said plurality of music 
apparatus, for wired input of audio signals or audio signal 
producing signals for producing audio signals that are output 
from the different music apparatus, wherein 

said mixing section also mixes the audio signals input by 
said wired input section or the audio signals produced 
on the basis of the audio signal producing signals input 
by said wired input section, in addition to the audio 
signals received by said wireless communication sec- 
tion or the audio signals produced on the basis of the 
audio signal producing signals received by said wire- 
less communication section. 

8. The mixer apparatus according to claim f, further 
comprising audio signal generating section for generating 
audio signals independently from said plurality of music 
apparatus, wherein said mixing sectional so mixes the audio 
signals generated by said audio signal generating section, in 
addition to the audio signals received by said wireless 
communication section or the audio signals produced on the 
basis of the audio signal producing signals received by said 
wireless communication section. 

9. A mixer apparatus comprising: 

a wireless communication section for receiving by wire- 
less first audio signals or first audio signal producing 
signals from a plurality of first music apparatus; and 

a mixing section for mixing the received first audio 
signals or second audio signals produced on the basis of 
the received first audio signal producing signals, 
wherein 

said mixing section mixes third audio signals or forth 
audio signals produced on the basis of second audio 
signal producing signals in addition to the received 
first audio signals and the produced second audio 
signals, when said wireless communication section 
receives by wireless the third audio signals or the 
second audio signal producing signals from a second 
music apparatus while said mixing section is mixing 
the received first audio signals or the produced 
second audio signals. 

10. A music apparatus capable of wireless communication 
with a mixer apparatus that mixes a plurality of audio 
signals, wherein the music apparatus comprises: 

mixing signal generating section for generating said audio 
signals that will be subjected to mixing or audio signal 
producing signals for producing said audio signals that 
will be subjected to mixing; 

a wireless communication section for transmitting by 
wireless to said mixer apparatus the audio signals or the 
audio signal producing signals generated by said mix- 
ing signal generating section and for receiving mixed 
signals mixed by said mixer apparatus and transmitted 
by wireless from said mixer apparatus, said mixed 
signals including said audio signals transmitted by 
wireless from said music apparatus or the audio signals 
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produced on the basis of said audio signal producing 
signals transmitted by wireless from said music appa- 
ratus; and 

reproduction section for reproducing the mixed signals 

received by said wireless communication section. 
11. A computer readable program applied to a mixer 
apparatus: 

said mixer apparatus comprising a wireless communica- 
tion section for receiving audio signals or audio signal 
producing signals for producing audio signals that are 
transmitted by wireless from a plurality of music appa- 
ratus, and mixing the audio signals received by said 
wireless communication section or the audio signals 
produced on the basis of the audio signal producing 
signals received by said wireless communication sec- 
tion, wherein 



said computer readable program allows said wireless 
communication section to function as a master and 
allows said plurality of music apparatus to function 
as slaves. 

12. A computer readable program according to claim 11, 
wherein 

said computer readable program allows said wireless 
communication section 

to operate to issue requests to said plurality of music 
apparatus for transmittance of said audio signals or 
audio signal producing signals, and 

to operate to receive said audio signals or audio signal 
producing signals that arc transmitted from said plu- 
rality of music apparatus in response to said requests 
for transmittance. 
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ABSTRACT 



A musical workstation system produces a display presenta- 
tion in one of a musical composition responsive to musical 
composition data and responsive to one or both of input 
variables and a selected operating mode. The system is 



comprised of (1) means to provide the musical composition 
data (such as local storage (ROM, RAM, CD-ROM, hard 
disk etc.), or via a communications interface to an external 
device (such as another music workstation, a master 
controller, a computer), a memory, a selection subsystem, a 
controller, and a display subsystem. The memory selectively 
stores the received original musical compositions. The 
selection subsystem determines a selected operating mode 
and display format. The controller, responsive to the selec- 
tion subsystem, provides means for selectively controlling 
the storing of the musical composition data in memory and 
selectively processing (e.g. altering) the stored musical 
composition data responsive to the selected operating mode 
and the input variables to produce a particular one of a 
plurality of processed results, such as external 
communications, operating, mode, transformation to deriva- 
tive musical compositions, etc. The music workstation can 
communicate with one or more external devices, such as 
other music workstations, a master workstation, a controller, 
etc. The display system provides for selection of original 
compositions, creation of derivative compositions, distribu- 
tion of compositions, monitoring of each performer's 
performance, group virtual performances, and also allows 
for local and distributed retrieval and editing of musical 
compositions, such as changing keys, pitch, tempo, and 
many other parameters. The music transformation can be 
performed locally or at the central or distributed music 
database, which stores the scores of musical composition. 
The musical composition data can be transposed via a 
controller, and can be transmitted to a plurality of the 
individual music workstations that then display the music 
composition, to permit playing by musical instruments or 
vocalists, and for location presentation. 

54 Claims, 27 Drawing Sheets 




U.S. Patent 



Jul. 4, 2000 



Sheet 1 of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 



Sheet 2 of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 



Sheet 3 of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 



Sheet 4 of 27 



6,084,168 




TIG- 22 



U.S. Patent 



Jul. 4, 2000 



Sheet 5 of 27 



6,084,168 




2C 



U.S. Patent 



Jul. 4, 2000 Sheet 6 of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 



Sheet 7 of 27 



6,084,168 




244 



•282 



Errors 



Communicate Errors 
To Display 



284 



(Audio +/- Video) 



286 



Generate Graded Score 
Indicating Errors 



No 




288 



T1Q. 



U.S. Patent 



Jul. 4, 2000 



Sheet 8 of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 



Sheet 9 of 27 



6,084,168 




U.S. Patent Jul. 4, 2000 sheet 10 of 27 6,084,168 



1 




U.S. Patent jui.4,2000 sheet u of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 Sheet 12 of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 Sheet 13 of 27 



6,084,168 




Master 
Workstation 


1 


1 
1 
1 


Master 
Workstation 


CM 



to 

S3 1 




U.S. Patent 



Jul. 4, 2000 



Sheet 15 of 27 



6,084,168 



804 



805 



806 



810 
820 
830~ 



Transmitter 



Music 
Database 



Main 
Controller 

kD — o 



Marching 
Embodiment 



850 



811 

r 

821 

831 



812 
822 
832 



840 



Receiver 



Controller 



Display 



841 



Receiver 



Controller 



Display 



t 



842 



Receiver 



Controller 



Display 



801 

Remote 
Unit#l 



_802 

Remote 
Unit #2 



-803 

Remote 
Unit #N 



8 



U.S. Patent jui.4,2000 Sheet 16 of 27 6,084,168 




m. 9 



U.S. Patent Jul. 4, 2000 Sheet 17 of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 



Sheet 18 of 27 



6,084,168 




U.S. Patent jui.4,2000 sheet 19 of 27 6,084,168 




U.S. Patent 



Jul. 4, 2000 Sheet 20 of 27 



6,084,168 



1150-^ 



Start of Networked Mode 



A Mode 5 



>dT> 



Selection of Composition 



Capture and broadcast of user performance 
data by a plurality of individual music stands 



Receives and combines the plurality of user performance data 
to generate combined performance data. 
Broadcast of combined performance data to all individual music stands. 



Audio Output of the Combination of audio performances 
of the plurality of individual music stands. 




JIQ. 12 



U.S. Patent 



Jul. 4, 2000 



Sheet 21 of 27 



6,084,168 



m- 13 




Compare 
Performance to Score 



I 1330 

i / 



Resynchronize Display of 
Music Score to real-time 
Performance Timing 




YES 



U.S. Patent 



Jul. 4, 2000 



Sheet 22 of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 Sheet 23 of 27 



6,084,168 




U.S. Patent 



Jul. 4, 2000 



Sheet 24 of 27 



6,084,168 




U.S. Patent Jul.4,2000 Sheet 25 of 27 



6,084,168 




if 



15 



U.S. Patent 



Jul. 4, 2000 



Sheet 26 of 27 



6,084,168 



NWSl 



NWS2 



NWSN 



Tl 



ra. 



ii- 



T2 



Individual Perfonnancc Data (EPD) 



Synchronize Virtual 
Performance Data 



Synch 

k- 'D -H 

J Recombines and pulls back 

into Synchronization 

(Max. Time Delay) 



a, b, c, d, e, f, = data and timing synch info 



Tty 18 



U.S. Patent 



Jul. 4, 2000 



Sheet 27 of 27 



6,084,168 



Master Station Buffer 



WS 1 


IPDAxxx 


WS2 


IPDBxxx 


WSN 


IPDNxxx 



Music 
-Data 
Buffer 
(w/i RAM) 



Combined Performance Data: Axxx + Bxxx + Nxxx = CPDXXX 



Individual Work Station 



Outgoing Live Performance Data: IPDxxxx 
Outgoing Combined Performance Data: CPDxxxx 



Music 
Data 
Buffer 
(w/i RAM) 



fig- 19 



6,0c 

MUSICAL COMPOSITIONS 
COMMUNICATION SYSTEM, 
ARCHITECTURE AND METHODOLOGY 

This is a continuation-in-part application, under 37 
C.F.R. §1.53, of pending prior application Ser. No. 8/677, 
469 filed on Jul. 10, 1996, for Multi-Dimensional Transfor- 
mation Systems and Display Communications Architecture 
for Musical Compostions, now U.S. Pat. No. 5,728,960. 

I. FIELD OF THE INVENTION 

The present invention relates to the field of music. More 
particularly, the present invention relates to a display system 
for displaying musical compositions, either batch or in a real 
time environment, and processing and communicating user 
performances. 

II. DESCRIPTION OF THE RELATED ART 
Music is usually only available in the written form for one 
(or a fixed set of) performer/instrument types in a fixed key. 
Adaptations or variations of musical arrangements are com- 
plex and costly. Remotely located musicians are unable to 
effectively practice together. Small communities each with 
only a few musicians are limited to practicing with the few 
musicians they have. 

Performers of music have many inconveniences to deal 
with. One such inconvenience deals with the composing, 
distribution, and utilization of music display presentation, 
traditionally sheet music. Another major problem relates to 
the inconvenience of scheduling and physical gathering of 
multiple musicians (including instrumentalists and 
vocalists), which when combined in their performance pro- 
vide a musical ensemble or orchestra. For example, high 
school band practice requires that all students lie available to 
practice at the same time at the same place (i.e., the school 
music room). However, this creates difficulties in that many 
students have other activities which conflict with band 
practice which is then incomplete. Additionally, when 
composing, musicians often will come up with an idea when 
physically not with another musician. 

Musicians typically work from sheet music. When 
composing, they write the notes down on paper that has a 
number of staffs. If the musician transposes a composition 
from one key to another, the notes are also written down on 
the staff paper. The scores for different instruments must also 
be generated and written down. All of the scores are then 
copied for distribution to other musicians and/or music 
stores. 

When performing, the sheet music must be found, for all 
parts to be played, manually distributed, manually set-up, 
manually handled (turn pages, etc.). There is also an unful- 
filled need for quick access to a more comprehensive 
database of music for the performing musician, whether he 
is solo or part of an orchestra. Also, musicians often perform 
audience requests, and require access to sheet music for 
requested songs. Presently, there are various combinations 
of songs compiled in "FAKE" Books, usually by category 
(e.g., rock, country, blues, big band, etc.). This is only of 
limited help. Furthermore, the use of paper sheet music is 
cumbersome and inconvenient; pages often get damaged or 
lost, and indexed access is poor and slow. 

This method of composing and distributing music is 
inadequate when the music is used by a band or orchestra 
that requires hundreds of copies. If the conductor desires the 
piece to be played in a different key or certain sections of the 
music edited to suit the conductor's tastes, the composition 
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must be rewritten and the new transposed copy distributed to 
the band or orchestra. This is a very costly, time-consuming, 
and laborious task if the orchestra has a large number of 
members. 

Additionally, if the composition does not have a part for 
a certain instrument, the conductor must generate the 
required part from the original composition. After the score 
for the required instruments has been generated, the parts 
must be copied and distributed to the individual musicians. 

10 This, again, is a very costly and laborious task if the band has 
a large number of musicians requiring different parts. There 
is a need, therefore, for a more efficient way of transposing, 
editing, and distributing music scores. 

Over the past many years, great advances have been made 

15 in the electronic input, storage, and display of music. Elec- 
tronic bands and orchestras are constructed using computers 
and MIDI equipment. Programs exist for personal comput- 
ers (e.g., Apple Macintosh, DOS, and Windows machines) 
for an individual to use the computer for transposing music, 

20 composing music. Programs also exists for automatically 
inputting music from direct performance (such as directly 
from a keyboard, electronically through MIDI converters 
(such as for string instruments), via pickups and 
microphones, and sequencers, tone generators, etc.) To gen- 

25 erate digital data and/or music notation. 

Musicians often perform both pre-planned and ad hoc 
compositions during the course of a performance. It would 
therefore be desirable to have the ability to access a large 

3n database of musical compositions on demand. It would also 
be desirable to permit communication and synchronization 
of a music presentation to multiple performing musicians 
who are playing together. It would also be desirable for a 
performing musician to have his or her performance of the 

35 music input onto an individual music workstation, and 
stored, and analyzed by an automated system, and/or com- 
municated to one or more other networked (remote) indi- 
vidual music workstations. 

SUMMARY OF THE INVENTION 

The present invention encompasses a musical presenta- 
tion system. In one embodiment, the system enables one or 
more users to select a musical composition, and perform the 
selected musical composition at each of a plurality of 

45 individual music stands, independently capturing the per- 
formance of a respective user and communicating the indi- 
viduals performance data to a master workstation (which can 
be standalone or one of the networked individual 
workstations), which combines the plurality of individual 

so workstation performance data into a composited combined 
performance data and communicates said combined perfor- 
mance data back to all of the individual workstations 
wherein the individual workstations provide for audio (and/ 
or vide and/or audiovisual) output representative of the 

55 combined performance data (which represents the musical 
performance inputs for all of the communicating plurality of 
individual workstations). The time between the users per- 
forming the segment of music and that same user hearing the 
audio presentation of the combined data is less than the time 

60 interval detectable by a human being. In a preferred 
embodiment, the plurality of individual workstations col- 
lectively provide for synchronized display presentation of a 
selected music composition's presentation, and provide for 
output of individual performance data representative of the 

65 musical performance of the user corresponding to the dis- 
play presentation. Timing-synchronization is also provided 
for to permit synchronization of the display presentation of 
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the plurality of individual workstations, and for synchroni- 
zation by the master workstation of the plurality of indi- 
vidual performance data to construct the combined perfor- 
mance data. In one embodiment, the master workstation 
generates a synchronization signal that is communicated to 5 
all the individual workstations from the master workstation. 
Other synchronization structures are also equally acceptable, 
such as embedding timing synchronization data within the 
individual performance data. 

In another embodiment of the present invention, a musical 10 
presentation system enables a user to select from one or a 
variety of musical compositions, control the adaptation of 
the selected composition, and distribut e the edited v ersion 
efficientl y in a paperless en vironment. The system and 
process of the present invention also provides me ans fo r 1 5 
recei ving music from a number of sources. The user' selects 
the "desired musical composition from the source. The 
desired composition is displayed and/or stored in the sys- 
tem's memory for further processing by the system prior to 
display and/or distribution. 20 

In accordance with one aspect of the present invention, 
each of the music workstations in an intelligent music 
composition communication architecture provides for musi- 
cal information, to be distributed for a video or visual 
presentation of music in a user-friendly notation, and/or 2 - 
provides an audio presentation that can be selectively 
tracked and synched to the video presentation and/or tracks 
and synchs the displayed video presentation to a live 
performance, etc. 

In still another embodiment, the system includes a user 
input device enabling selection of the musical composition, 
and optionally, permitting any user specified editing desired 
in the composition, and, in a preferred embodiment, permit- 
ting user selection of parameters (such as the musical key in 
which the composition is to be played). The user can then 
instruct the system to transmit the newly generated music 
scores to one or more display subsystems (such as CRT's, 
LED's, LCD's, etc.), or to other systems. In the preferred 
embodiment, these displays take the form of liquid crystal 4Q 
displays built into music stand based systems, also referred 
to herein as display stations or workstations. 

This invention also relates to a musical presentation 
and/or communication system, and more particularly, to a 
system which permits musicians to view the score or other 45 
audiovisual or visual-only presentations of a musical 
composition, to permit the musician/user to perform the 
musical composition on the instrument of choice, as selected 
by the user, and in the key of choice, as selected by the user. 

In accordance with one aspect of the present invention, 50 
one or more music workstations are provided, consisting of 
a video presentation means, such as a display screen (CRT, 
LCD, LED, Heads Up Display (HUD) etc.) in conjunction 
with a computer-based system which in one embodiment 
stores a database of songs and music which can be utilized 55 
by the musician/user of the system. 

In accordance with yet another embodiment of the present 
invention, there are provided numerous music workstations, 
such that different or similar instruments can each select a 
respective portion of a song to be performed, such that all 60 
musicians performing the same musical piece are provided 
musical presentation or representation in the same key to 
permit the playing of the music together. In a preferred 
embodiment, the system is capable of locking into a par- 
ticular key (and/or instrument type) responsive to input 65 
variables (such as can be provided via an input device such 
as a microphone or keyboard, or voice recognition, or 



camera) to determine the desired key to be played in. In a 
preferred embodiment, the user can select an auto-transpose 
mode where all music is thereafter automatically transposed. 
The transposition can take place at a master workstation and 
then be communicated to the individual workstations. In one 
of the illustrated embodiments, the transposition of the 
music takes place locally at the music workstation which 
provides local intelligence. An original musical 
composition, such as selected from a stored database of a 
library of music is then transposed in accordance with 
transposition rules. Many options are available, and various 
transposition rules and methods are well known and docu- 
mented and in use on numerous products, such as keyboards 
which have built-in capabilities for transposition of music, 
as well as computer software which provides for transposi- 
tion and notation of music. 

In accordance with an alternate embodiment, the user 
connects to a remote database via wireless or wired com- 
munication to permit downloading to the local computer of 
those musical pieces which the user has selected for perfor- 
mance. The user music terminal, or the central computer, can 
receive the transposed (derivative composition) version or 
can receive an unmodified (original musical composition) 
version, which it can then display and/or convert and trans- 
pose music as necessary for each instrument and/or key. 

In another alternate embodiment, a user can prearrange 
for downloading of selected compositions via a remote 
connection service, where the music user terminal can 
include a non-volatile storage memory permitting the stor- 
age and replay of all requested compositions. Alternatively, 
a central server can be provided, where multiple music end 
terminals share a single central controller computer, or each 
have their own computers which share in a central server 
computer or with each other in a distributed architecture. 

Alternatively, a non-volatile storage structure, such as a 
CD-ROM, can contain a database of music which can be 
selected from for transposition in accordance with the 
present invention. 

The present invention provides a music communication 
architecture and methodology that permits the synchroniz- 
ing of music display presentations for multiple display 
stations performing the same musical composition. Bands 
and orchestras can be constructed using multiple indepen- 
dent music workstation display whether at one site or 
remotely distributed. In one embodiment, the music display 
presentations provides one or more of the display of the 
musical composition, performance by the user, and the 
output presentation for the combination of the individual 
performance data for a plurality of individual workstations. 
In one embodiment, a master workstation is responsive to 
the individual performance data from the plurality of indi- 
vidual performance data and provides means for synchro- 
nizing and compositing the individual performance data 
from the plurality of the individual workstations, and pro- 
viding a presentation output (audio and/or video) comprising 
the combined virtual performance. 

It is a further object of the present invention to permit the 
comparison of a performer's performance parameters, such 
as parameter signals obtained via a microphone and/or 
camera, of the performing artist's performance or execution 
as compared to the stored and displayed music. The com- 
parison includes the pitch, timing, volume, and tempo etc. of 
the music (such as through audio recognition) and critique 
the artist's physical movements (e.g., proper finger position, 
etc.) through visual recognition. In a preferred embodiment, 
the music workstation system provides the performer and/or 
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a conductor with a presentation performance feedback indi- compositing the plurality of individual performance data 

eating the quality of the performance as compared to the signals which virtual performance data 1451 is communi- 

stored or displayed music, such as any errors, where they cated back to all of the plurality of individual workstations 

occurred, etc. 1410 which provide audio output responsive thereto; 

It is a further object of the present invention to provide a 5 FIG. 15 illustrates one example of data flow communi- 

system whereby any displayed music can be transposed cation architecture for a plurality of physically separate 

(e.g., to a different key, or converted to include different or remotely located locations, such as a teacher/master 

additional different instruments and voices than that in conductor/server location and each of a plurality of locations 

which the sheet music is originally displayed). that provide communication of performance data for users of 

It is a further object of the present invention to provide 10 the individual workstations at each of those locations, 
automated modes of intelligent operation of the music wherein timing information is utilized by the master work- 
workstations, and to provide responsiveness to multiple station to reconstruct and combine all of the individual 
forms of user input performance data into a combined virtual performance data 

These and other' aspects and attributes of the present „ ° ut P ut c ° u P led back to the P"' '° f individuals worksta- 

invention will be discussed with reference to the following 15 tlons whlch res P onsive P rovlde audl ° 0Ut P Ut; 

drawings and accompanying specification. FIG - 16 illustrates a data word structure compatible with 

one embodiment of the present invention; 

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 17 illustrates a block diagram for a workstation 

FIGS. 1A and IB show a music presentation system in ,„ (g enera "y shown in FIGS. 5-7) and further comprising a 

accordance with the present invention; network interface and a MIDI performance data processor, 

FIGS. 2A-2G show flow charts of the processes in FIG. 18 illustrates a timing diagram showing the synchro- 
accordance with the present invention; nizat i° n timin g of the architecture as illustrated in FIGS. 14 

FIG. 3 shows one embodiment of the display for the and 15 ' and data stru , cture illustrated in FIG. 16, and 

music display workstations and input devices in accordance * showing the composited I synchronized virtual performance 

with the present invention; data and lhe multl P le individual P^nnance da ^ a ° d 

FIG. 4 shows a shared music database and stand alone FIG - 19 f illustrates one example of a memory work space 

workstation embodiment in accordance with the present slructure for lhe uldividual and mas er workstations in the 

invention- virtual performance mode, in accordance with the present 

' , . . . 30 invention. 

FIG. 5 shows a music communication system in accor- 
dance with the present invention; DETAILED DESCRIPTION OF THE DRAWINGS 

FIG. 6 shows a master workstation and slave workstations 

in accordance with the present invention; While this invention is susceptible of embodiment in 

FIG. 7 shows an alternate embodiment of the present 35 many different forms, there is shown in the drawing, and will 

invention using one or more of the workstations coupled to be *«nbed herein ,n detail, specific embodiments thereof 

a master controller and music database; Wlth . tbe understanding that the present disclosure is to be 

„ , . • j considered as an exemplification of the principles ot the 

FIU 8 shows a marching band environment in accordance invcntion aQd . & m{ , o ^ , h / invcmion t0 the 

with the present invention; embodiments illustrated. 

FIG. 9 shows a person outfitted with a sensor body suit in 40 ...... u- r.u 

, .,, v .... In accordance with the teachings ol the present invention, 

accordance with one aspect of the present invention; , . ... , a At 

„ „ , 1 . . a system and methodology are provided tor music presen- 

FIG. 10 shows a movement and pattern recognition sys- , ation and comraunication . Musica i compositions can be 

tern in accordance with one aspect of the present invention; inpu( (0 the presenl invenlion frora any one or more of 

FIG. 11A illustrates a flow chart for an alternative multiple sources, such as from prestored score images, live 

embodiment of the operation of the automated mode "A ' m i cr0 phone, direct input from musical instruments or vocal 

Mode" 240 of FIG. 2A, alternative to that illustrated in FIG. dircc , performances, scanning in of existing printed score 

2Q images (optically character recognized), cameras, visuals, 

FIG. 11B illustrates the process flow for an alternative etc. These inputs by the system are used in the selective 

overall operation relative to FIG. 2A of a music composition 5Q storage, composition, communication, and presentation of 

communication workstation in accordance with the present the musical system of the present invention. The system can 

invention, wherein steps 200-250 of FIG. 2A are redefined generate additional material automatically, or permit a user 

to show an overall operation which also encompasses a to modify, communicate, display and/or reproduce the musi- 

specific networked mode A Mode 5, as further illustrated in cal compositions. 

FIG - 1 2 ' 55 Modification can be performed on rhythm, primary key, 

FIG. 12 illustrates the process flow for the networked individual notes, chords, etc. The vast_^ store_of_ musical 

virtual performance mode 1150 of FIG. 11B; inforn^ion_ stored in digital notat ion formaTand/or any 

FIG. 13 illustrates the process flow for automated mode 6 video format, can be broadcast (analogjjrdigital) to a local 

corresponding to process step 1153 of FIG. 11A, directed to music workstation or a master KmTraiTer, which can also be 

synchronizing the display to the performance, to resynchro- go a local workstation. The master controller can be a stand 

nize the display of the music to account for performer error; alone unit, or act as a server as well as its own stand alone 

FIG. 14 illustrates a communications architecture in unit, or simply as a server to a plurality of other stand alone 

accordance with the present invention, where each of the units. However, in the minimal configuration, only a single 

individual music stands 1410 receives a music input 1411 musical user station is needed. 

and provides performance data output 1412 coupled to and 65 In one preferred embodiment, the workstation is provided 

communicating with a master workstation which generates as a music stand where the display presentation is a liquid 

an output of virtual performance data 1451 responsive to crystal display (LCD). The LCD that can provide 
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monochrome, gray scale or high quality color displays, 
depending on design and cost constraints and desires. Other 
display types can also be used. A touch-screen input can 
provide for simplified and adaptive user input selections. An 
optional built-in metronome function can also be provided 
for display presentation audio and/or video. A subsystem can 
also be optionally provided to permit the music to be audibly 
reproduced at the workstation through a speaker or head- 
phone jack or other output. 

It is well known in the art to convert user analog audio 
input into a digital format, ranging from straight Analog to 
Digital (e.g., A/D) conversion to processed data conversion 
to encoded digital music data, such as MIDI. Examples of 
MIDI include guitar input or other stringed instrument input 
through microphones or directly to MIDI-converters, or 
voice/non-pickup instruments through microphone con- 
verted to MIDI-inpul, or keyboard MIDI-input. Such input 
systems are commercially available from numerous compa- 
nies for numerous types of interfaces at numerous interface 
levels. Similarly, numerous A'D converter subsystems are 
commercially available at chip and board solution levels 
(such as from Analog Devices Corporation and from Mat- 
trox Systems). 

In accordance with one aspect of the present invention, 
the multi-dimensional music transformation system of the 
present invention also enables a user to select one or more 
musical compositions from a larger database from a plurality 
of musical compositions. The database can be stored locally 
within the workstation, on site, or remotely stored and 
transmitted to the user (such as over cable, wire, telephone 
lines, wireless (such as radio frequencies)). The user can also 
optionally edit the selected score of the composition (such as 
changing the key and/or any note and/or timing, etc.) to suit 
his or her taste. The score (altered (the derivative 
composition) or not (the original composition)) can then be 
transmitted to one or more displays such as liquid crystal or 
CRTs in the music stands of the band or orchestra. The 
present invention, therefore, provides an efficient, paperless 
solution to communicating, presenting (displaying), and 
optionally one or more of transposing, editing, inputting, 
comparative testing-teaching, conducting, and disseminat- 
ing music to one display or a large number of displays. Each 
display can have the same, or a different, unique, customized 
presentation of music notation as appropriate per selection, 
responsive to a set-up by a system, automatically per pre- 
defined parameters, and/or to user input. The score can also 
be printed out if a hard copy is desired. 
^AS illustrated in FIG. 1, the music is stored, such as on a 
,4arge hard drive or CD ROM jukebox, in a digital format as 
a music library (120). The music library (120) is coupled to 
a processor subsystem (115). Coupling can be wireless or 
cabled such as through a shielded cable, fiber optic 
conductor, switched connection (such as via phone lines), 
local, or remote. The processor (115) has the local storage 
capacity (e.g., semiconductor memory, disk storage, etc.) to 
hold the digitized version of the music composition trans- 
mitted to it on request from the library (120). The music 
library can be local or proximately remote from the rest of 
the system. 

In a wireless embodiment, the music library (120) is 
coupled to a communications subsystem (such as a radio 
frequency transmitter) (125) that transmits the contents of 
requested compositions from the remote music library (120) 
to the proc^ssor""(Tl5) througTTTte^nTenna (104) of the 
transmitter. The antenna (102) at the receiver picks up the 
transmitted signal and the receiver conveys it to the proces- 
sor (115). This embodiment enables the music library (120) 



to be remote and located at a great distance from the 
requesting site. The communications subsystem (125) can 
be a transceiver for bidirectional wireless communication, or 
a transmitter for one-way wireless communication (such as 
where the requests are otherwise communicated to the music 
library subsystem (120), such as via a wired connection). 

As illustrated in FIG. 1A, a system controller, in the form 
of a music stand (105C) with a liquid crystal display, is used 
by an operator (e.g., performer, conductor, etc.) to select one 
or more musical compositions. FIG. 1 A illustrates two types 
of music workstations stands. The workstation stand (105C) 
provides certain optional features foTa more tuTl-featured 
stand, including as illustrated, speakers (140) both wireless 
and wired communications capability, and as illustrated, 
shows the processor with memory (115) as an external 
separate component. The music stand (105P) shows the 
integration of the proce ssor and m emory into the music 
stand itself, and alscTsnows both wireless (antenna (101)) 
and wired connection (port (107)) to permit network com- 
munication. Alternatively, the conductor stand (105C) could 
have all or part of the features integrated into the music stand 
(105C). Depending on the function for which the music 
workstation stand will be used, some or all of the features 
can be provided for that stand to minimize costs or optimize 
versatility. For example, in one situation, only the teacher or 
conductor needs the full-featured, full-powered music work- 
station. In that-case, the performers or students do not have 
a full-feature workstation, but rather a scaled-down version 
of the workstation stand. In the preferred embodiment, a 
user input device (110) (such as a touch screen, microphone, 
keyboard, switches, voice recognition system, visual recog- 
nition system, etc.) is coupled to the processor in a wired 
(such as over a cable or fiber optic link) or wireless (such as 
over an RF link or infrared link) manner for workstation 
stand (105C), or directly to the processor, where it is built 
into the system controller as workstation (105P). The user 
can select an original musical composition from th e" touch 
screen of the liquid crystal d isplay (135). The processor 
responds by storing that composition in the memory (115) of 
the local workstation of the user as requested. 

Using the touch sensitive LCD (135), the user c an now 
create a deriv ative musical composition . The touch sensitive 

LCD ail; 



"the user to enter the musical key in which the 
original composition will be played, e dit any notes desired, 
and selecnhejnslrumenls and parts that will be playing the 
composition. The composition as originally composed, and 
the derivative or modified composition can be played back 
to the user over speakers (140) so that he or she may listen 
(e.g., such as to observe how the changes will sound) while 
optionally permitting simultaneous viewing of the score on 
the presentation visual display. Once the score has been 
designated (e.g., selected, edited, etc.) to the users (e.g., 
conductor's) taste, the appropriate portions (e.g., by musical 
instrument) of the scores can then be transmitted for 
(optional storage and) display to the respective associated 
individual music workstation stands of the band members. 

In a preferred embodiment each stand has an input device 
(110) that permits the user of the stand to select which 
instrument will be using the stand. (As discussed above, this 
input device can take the form of a touch sensitive screen or 
a number of buttons or switches or voice or audio 
recognition, etc.) 

In the preferred embodiment, each individual music 
workstation stand (105) can be directly and/or remotely 
programmed to addressably receive (and optionally to 
locally convert) and display the music score that is intended 
for the respective instrument type (user type) that will be 
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using (is associated with) the stand. As an example, the user 
of the stand (or a conductor) can input their selection of 
saxophone into the user input device (110) of the worksta- 
tion stand (105C), to progra m that workstation st and (105C) 
onl y to receive the musical score for the saxophon e (see 5 
FIG - ! - J). Then, the musical scores for all selected parts can 
be independently broadcast to all connected workstation 
stands, with each individual workstation stand individually 
distinguishing' and accepting only Its part. Alternatively, 
each workstation stand can be in3ivi3ually addressed for 10 
separate broadcast reception of its own respective selected 
part. Additionally, the user of the stand can program the user 
input to select a musical part of a selected musical compo- 
sition (e.g., saxophone first chair) and receive only the 
musical score intended for that chair. Tfiis same procedure 15 
can be followed for other instruments within the band or 
orchestra. Alternatively, a single music composition can be 
broadcast to all workstations, where "each workstation has 
local intelligence (processing and storage) to p ermit loc al 
conversion for display at each workstation for th e selected 20 
instrumenl for eacB workstation. 

For wireless communications, the individual music work- 
station stands (105) are comprised of receivers (or trans- 
ceivers where bidirectional communication is desired) and 
antennas (101, 103) for receiving (or transceiving) the radio 25 
frequency information from (and to) the master workstation 
(such as for the conductor). The music stand also has a 
display (such as an LCD (135)) for displaying the musical 
score intended for that stand. 

Referring to FIG. IB, the music wor kstation sta nds can ' ,0 
either JjnidcjQl ical or brok en d own into conductor stands and 
performer stands. A conductor stand (105CON) may have 
more functions and control than a performer stand 
( 105PLR). Aperformer stand (105PER) might only have the 
ability to receive and display musical scores, whereas the 35 
conductor stand (105CON) has the ability to select the 
musical score, change the key of the musical composition, 
and perform other tasks only a conductor would be permit- 



$-0^ jJUed or required to do. 
^iVf In 



one embodiment, an RF antenna for the stand (105) can 
be built into the stand itself Alternatively, instead of using 
RF, the performer's stand can be linked to the main (e.g., 
conductor's) stand using infrared, fiber optic cable, shielded 
cable, or other data transmission technologies. As discussed 
above, the communications link can be bidirectional, such as 
to facilitate requests and responses to facilitate the feedback 
of performance parameters or such that any workstation can 
be a master or slave, or used in combinations. 

FIG. 2A illustrates the overall operation of the music 
composition communication workstation. It begins by start- 
ing up the system (200). The system then provides a menu 
(201) that allows the user to select a listing of available 
music compositions. The user then selects one or more 
compositions (210). If the user selects one from the menu 
that is locally stored, it directly retrieves the information. 
Alternatively, if it's not something locally stored, the system 
couples (e.g. will dial up or go through a database or 
network) to a remote storage site and requests and receives 
the selected compositions. 

Any changes that are desired to the composition can be 
selected at the next logic block (215). If there are changes 
(such as to the key, or note editing, or selection of form of 
display or instruments), then those can be accomplished as 
illustrated at blocks (255) to (285). 

If no changes are desired, the musical score for the 
composition that is selected is broadcast, transmitted, or 



otherwise transferred to the workstation music stand (220). 
It is internally stored in the local workstation music stand. 
Next, the score is displayed (225) on the workstation display 
(e.g., LCD or CRT) or a video projection system. The 
display can also be part of an integral stand-alone worksta- 
tion or an interconnected group of components including a 
personal computer (such as Macintosh, or DOS or Windows 
PC). 

The display mode selection is then made (230). This 
permits selection of an operational display mode, not simply 
choosing the resolution or color. The two main choices in the 
preferred embodiment are a manual mode (250) and an 
automated mode (240). In the automated mode selection 
(240), there are many sub-modes or options, such as the 
operational mode that permits the performer or user to do 
their performing without having to tend to the selection of 
the portion of the music to be displayed or the turning of 
pages. In the auto performance mode as shown on LCD 
(135P), there is provided the simultaneous displaying of the 
selected musical composition, and a display representative 
of the audio performance of the user, and a divergence signal 
or divergence data representative of analyzing the 
performance, preferably in approximately real-time. 

FIG. 2B illustrates the manual mode (250), which pro- 
vides for user manual selection of functions (252). There are 
many functions that the user can select, even in the manual 
mode, such as hitting a button or a touch screen to cause the 
turning of the page of the display. Another function is to go 
back a page or to scroll forwards or backwards. For those 
who are vision impaired, another function can increase the 
font size of the music presentation. 

Thus, there are many manually selected functions that can 
be provided. While the manual mode can have automated 
functions selected, it is distinguished from the automated 
mode where control is partially pre-defined without user 
assistance. In the manual mode (250), the user selects any 
and all features that are going to be provided (some of which 
can be automated). The selected function is then processed 
(256). 

Next, any ongoing needs are processed (257). These 
needs can include any overlapping automated function (not 
otherwise inconsistent with any other selected function). 

Referring to FIG. 2C, the operation of the automated 
mode "A Mode" (240) is illustrated. First, the user selection 
of the desired automatic mode is detected and responded to, 
illustrated as the auto-advance mode (242), the training 
mode (244), the performance mode (246), or any one of a 
number of other modes (248) as is described in further detail 
hereinafter. For example, auto repeat mode can be selected 
by designating the start and stop points, and the number of 
times to repeat a "looped" portion (or portions) of the 
displayed musical composition. Marching band mode (auto- 
advance based on metronome function, conductor control, 
etc), auto-compose mode, and many others can also be 
implemented. The order of selection of auto-advance, 
training, or performance mode is arbitrary, and the user can 
alternatively decide from a menu where all are simulta- 
neously presented as choices. 

The display can advance the music by page option, or by 
a user selection of one of many options (e.g., scrolling, 
tablature, video graphic tutorial display, etc.). 

Referring to FIG. 2D, the automated mode 1 for auto- 
advance operation (242) of FIG. 2C is illustrated, where the 
user has selected an auto-advance performance mode. In this 
mode "A Mode 1" (271), the system tracks the performance 
by the user of the composition to the score (272). Perfor- 
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mance refers to the actual performance by an individual 
person (or people) who is (are) reading the musical score 
upon which the performance is based. Whether that score is 
in tablature formal, staff and clef and note notation, or some 
other format, the system generates appropriate signals to 
permit comparison of the user's performance to the musical 
score. 

Based on a comparison, a decision is made pursuant to 
selection criteria programmed into the system (such as the 
rate at which the piece is being played, the time signature, 
the tempo, the rhythm, and the advancement of the music on 
the available display), the display presentation is advanced 
(274 and 278). In some cases, the music might move 
backwards, such as with D.S. Coda. The presentation of the 
display tracks the performance to permit smooth, uninter- 
rupted playing or singing. The capability can be provided for 
the user to over-ride this auto-advance, such as for practicing 
where it is desired to keep going back over sections. In this 
case, a user over-ride option (276) is permitted to alter the 
automated operation. Upon cessation of user over-ride, the 
system can be programmed to stop, to automatically return 
to the regular auto-advance mode, or to process other 
auto-modes (270) of FIG. 2C. 

Referring to FIG. 2E, the automated mode "A Mode 2" 
(244) operation of FIG. 2C is illustrated corresponding to the 
training mode. In this mode, the system tracks the perfor- 
mance (280) of the individual user to the composition score, 
primarily for the purpose of permitting a critical analysis and 
comparison of the performance to the score (282). This 
analysis determines divergence from the selected musical 
score, and reveals errors or deviations from desired perfor- 
mance goals (e.g. match of timing of notes, duration of 
notes, pitch of notes, etc.), and to display those errors (284) 
(such as by audio or video means). Predefined performance 
goals provide the knowledge basis for expert system based 
analysis. 

'ITie system can then generate a graded score (286) 
indicating errors, and can present it in numerous formats 
such as histograms, frequency of errors, spacing of errors, 
etc. Identification of when the errors occur (e.g., only when 
going from slow to fast, or fast to slow), absolute position 
within the score and so forth, are also tracked and reported. 
Other expert system rules can be provided by music teachers 
which give the necessary parameters for modeling expert 
system reasoning, as well as guidance and suggestions on 
how to correct problems such as via display text, graphics, 
audio, etc. 

The comparison of the performance to the score in the 
training mode is for the purpose of detecting the performer's 
compliance to parameters (such as the tempo, rhythm, filter, 
parameter, pitch, tonality, and other features that are adapt- 
able or can be modified by performers). This parameter 
information is available and published in numerous forms. 
Thus, having provided this core set of parameters, the 
system can thereafter perform the training automated mode. 

As illustrated in FIG. 2F, automated mode 3 "A Mode 3" 
is the performance mode (246). In this mode, the operation 
is as in automated mode 1 (auto-advance mode) except that 
no user over-ride is permitted. Its primary purpose is to 
accompany the performer during the entire performance of 
a score as an automated page turner. The tracking of the 
"page turning" to the performance can optionally be based 
on inputs or criteria independent of a performer's actual 
performance input (e.g., microphone), such as a built-in 
metronome clock, a central control (e.g., a conductor or 
special user input), etc. Additionally, performance charac- 
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teristics can be tracked, computed, and reported as in the 
teaching and training mode. Training feedback can option- 
ally be provided real-time, or subsequent to completion of 
performance, to assist the performer as in the training mode. 

5 Alternatively, the score can be presented in a moving score 
mode (e.g., vertically, horizontally, or otherwise) or linear 
presentation as opposed to a real page turning display. 

FIG. 2G shows the operation of automated mode 4 ("A 
Mode 4") which provides for the processing of other auto- 

10 mated functions selected by the system. These modes can 
include conductor mode, karaoki mode, etc. 

In conductor mode, a conductor can control communica- 
tions of signals to his or her performer (such as "increase 
volume", or "increase tempo", or "play staccato"). Icons can 

15 be provided where the conductor simply touches a touch 
screen (or other input mechanisms) to supplement his hand 
and body motions to permit more effective communication 
with the performers. Alternatively, as illustrated in FIGS. 9 
and 10, in a more advanced system version, the conductor's 

20 movements are first learned by a monitoring system, based 
on user definition and assignment of meanings for move- 
ment to provide an expert knowledge database. 

This system provides for tracking of movement input such 
as in FIG. 10 via video camera (1005) input of the conductor 

25 (1015) against a backdrop (e.g., blue screen) (1010) is 
processed by video processing unit (1020), or, as shown in 
FIG. 9, via body glove technology (gloves (935) or sensors 
(944) or sensor clothing (940) or head or eye movement 
tracking sensor (930) (such as used in virtual reality, High! 

1(1 simulation, avionics equipments (such as jets and space 
travel), and sports players for analyzing movement) to 
provide the necessary movement input. This movement 
input is analyzed utilizing the expert knowledge database to 
automatically generate a display (video and/or audio) to 

35 provide local visual and/or audio reinforcement on the local 
display (such as overlaying on a then unused portion of the 
music score display as a picture in a picture) to permit audio 
and video reinforcement of the conductor's body language. 
Thus, "a hush" body language signal that is directed towards 

40 a particular section of the orchestra would automatically be 
interpreted to cause the system to indicate, and only on that 
particular section's respective displays, a message (e.g., big 
face with a finger in front of it making a hush sound with a 
"hush" sound simultaneously output from a speaker). The 

45 conductor mode provides many benefits to performance and 
communication. 

For all these automated modes (e.g., A Modes 1, 2, 3, 4), 
training feedback can be provided real time or subsequent to 
performance at either or both of the perforner's workstation 

50 and a second (e.g., teacher's) workstation. 

The advantages of electronic music composition, com- 
munication and display are many. In addition to those 
discussed elsewhere herein, a capability exists for expert 
system based artificial intelligent type assistance where the 

55 expert system assists in many of the functions performed in 
musical composition and performance. For example, in the 
Auto-Compose Mode, if the words need to be changed to 
match the meter, equivalent terms can be chosen from the 
many sources such as thesaurus, dictionaries, rhyming 

60 dictionaries, encyclopedias, etc., to assist as well. Phrases 
from poetry, selected and indexed by content or topic can be 
re-expressed to create new works. Drum and rhythm section 
accompaniment can be expertly suggested, as well as 
harmonies, melody lines to accompany chords, chord pro- 

65 gressions to accompany melodies, harmonies to accompany 
a melody, and suggested musical instrument groupings to 
support a particular sound, rhythm, style, tonal quality, etc. 
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The expert system can be built from commercially avail- 
able technology, including component hardware systems 
with supporting software, as well as commercially available 
software packages which operate on commodity-type per- 
sonal and business computers such as the Macintosh by 
Apple Computer, Windows and DOS machines based on the 
X86 and Pentium processor technology of Intel, technology 
based on the Power PC and 68 XXX processor by Motorola, 
DEC PDP-11 technology, Sun workstations, etc. Custom 
microcomputer or DSP based system architecture on a chip 
can also be constructed, as well as ASICs, custom or 
semi-custom logic. 

'ITie system can be designed to take advantage of expert 
system design knowledge. A database of rules and facts are 
provided, and accumulated over time by the system in a 
sclf-lcarn mode. The expert system itself has the necessary 
logic to probe the user, monitor the performance, and apply 
the rules to provide feedback and reports to the user of skill 
level, errors, automated performance display, etc., starting 
with a base defined set of rules, instructions, and a knowl- 
edge database specific to music. 

The form of the musical score communication can be 
easily shaped to fit needs. One example is MIDI (Musical 
Instrument Digital Interface standard) which has advantages 
such as of bandwidth of storage used, is widely available 
commercially, is standardized, etc. However, signal 
processing, text, icon-based, object based, and various other 
forms of storage, user interface, and processing can also be 
applied to more specific applications of product. 

FIG. 3 illustrates one embodiment of an LCD display used 
for input control and for displaying the information from the 
processor and memory. In the preferred embodiment, this 
LCD is a touch sensitive screen enabling the functions 
associated with each displayed button to change, and also for 
the displayed buttons to be moved around the screen, 
depending on the function to be activated. The musical score 
may be edited by the conductor, such as by touching the 
individual note after which he is presented with a number of 
notes to replace the touched note. The lower portion of the 
screen displays instruments from which the conductor can 
select which instrument will be playing the composition. 
After a button on this screen has been touched, a number of 
sub-screens may come up, each with their own individual 
touch sensitive areas and functions to be activated by those 
areas. Alternatively, in addition to or instead of the touch 
screen, the system can provide input via separate key 
switches, voice recognition, etc. 

As an example, if the conductor touches the transmit key 
on the main screen, he will be presented with a screen 
showing all of the instniments that he has selected for that 
piece and a button labeled "ALL". He may now transmit to 
each individual music stand or by depressing the "ALL" 
area, transmit to the entire orchestra. 

The music library can be contained ("stored") on non- 
volatile storage either locally or at a remote central site 
containing the entire (or a subset) database of all possible 
music (that is then downloaded to local storage on request, 
either real-time at performance time or in advance.) 

Alternatively, the music library can be provided on stor- 
age medium that can be easily transported and used on site 
locally with the presentation system. Thus, for example, disk 
drives, cartridges, FLASH RAM cards, plug-in memory 
modules, or a CD-ROM or multiple CD-ROMs in a 
CD-ROM changer can be used to store and contain massive 
data libraries on musical compositions. While this would be 
a more expensive route than shared use of a central library, 
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requiring each musical group to obtain libraries on all 
possible compositions they may want, it has the advantage 
of speed, flexibility, no need for communication with a 
separate remote source, and creates a whole new mass 

5 marketing area (such as for CDs or Digital Audio Tape 
(DATs)). Another way of utilizing this technology is to 
maintain a history of music used, either with the remote 
music library or local music library. This could be done for 
many reasons, including copyright royalty assessment, 

to determining a history of musical performances and requests 
for future use in determining performance itineraries, etc. 
Alternatively, a hybrid of locally stored and centrally shared 
libraries can be utilized to optimize cost, speed and flex- 
ibility benefits. 

1 5 In accordance with another aspect of the present 
invention, each display workstation can also provide the 
ability to convert performed musical compositions into 
annotated musical compositions, generating the appropriate 
musical notation (e.g., staff, tablature, MIDI), notes, time 

20 signature, key, instrument, or user type, etc. 

The display workstation can be implemented as a totally 
self-contained workstation, where each workstation contains 
its own processing sub-system, optional communications 
interface (such as wireless or cable) for network use, input/ 

25 output interface including one or more of a user input 
keypad, a speaker, a microphone, joysticks, push buttons, 
etc. Each of the stand alone workstations can then operate 
with a local database or couple to a shared music database 
as illustrated in FIG. 4. 

The stand alone workstation(s) (105), are coupled to the 
shared database interface (405), and can either couple 
remotely (e.g., via phone lines) to the remote shared music 
database or to a local shared or dedicated music database 
(410). The shared music database (410) can either be pri- 
marily a storage means (e.g., hard disk or CD-ROM), or can 
include a processing sub-system (420) for local intelligence. 
In one embodiment, the stand alone music workstation 
includes the shared music database (410) and interface 

^ (405), non-volatile local storage medium for the shared 
databases (410), and a local processing subsystem (420), and 
can operate completely stand-alone. In an alternate embodi- 
ment of this stand-alone device, the shared database inter- 
face is contained in the stand-alone workstation (but not the 

45 shared music database or processing subsystem), and pro- 
vides capability for communication with a stored database 
(410) remote from the stand-alone device. 

In cither of these embodiments, an alternate additional 
embodiment provides capability for each stand-alone work- 

50 station to function as a master stand-alone, or a master or 
slave workstation within a workstation set including mul- 
tiple stand-alone workstations, wherein one is designated 
master and the rest are designated slaves. The slave work- 
stations in this configuration receive communication of 

55 music compositions to be displayed from the master 
workstation, thereby permitting one shared music database 
to be communicated among all workstations which are a part 
of the group. It is to be appreciated that the shared music 
database function can be distributed in many different ways 

60 among the workstations, or separable from and independent 
from the workstations. The choice is simply one of design, 
and the illustration herein should not be taken in a limiting 
manner. 

In one embodiment the master workstation has complete 
65 control over the slave workstation. Anything displayed on 
the master workstation is also displayed on the slave work- 
station. It is also possible for the user to mask certain 
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portions of the display of the master workstation before it is 
displayed on the slave workstation. In this manner, the 
conductor, using the master workstation, can transmit to the 
slave workstations only that information that is required by 
the orchestra members. 

In an alternate embodiment, the slave workstation com- 
municates performance parameters or deviation signals to 
the master workstation, for error analysis feedback. 

In accordance with another aspect of the present 
invention, moans arc provided wherein a plurality of indi- 
vidual workstations are coupled together in the network 
configuration to provide for networked communication of 
musical performance data wherein each of the individual 
music workstations provides for capturing the performance 
data for a users performance and communicating that per- 
formance data to a master or a conductor workstation which 
synchronizes and combines the plurality of individual work- 
stations performance data to create a composite virtual 
performance data output which is recommunicated back to 
all of the individual workstations in approximately real time, 
so that the individual workstations can receive the composite 
virtual performance data and provide an audio output (and/ 
or visual presentation) of the combined composite virtual 
performance data including all of the individual worksta- 
tions users performances. While the networking can be used 
in conjunction with other features and embodiments of the 
present invention, including communication of musical 
compositions for display, and other musical performance 
data analysis and data communication, as well as inter- 
musician interpersonal communication, the networked 
embodiment of the present invention permits synchronized 
virtual performance thereby permitting multiple remotely 
located individual workstations to physically separately per- 
form with the benefit of hearing in approximately real time 
the combined result of all the performances at the individual 
workstations with their own individual performance. 

In accordance with another aspect of the present 
invention, means are provided to permit a user of the music 
workstation to accomplish a transposition of a musical 
composition in pitch, tempo, and otherwise. In a preferred 
embodiment, the lead voice or instrument can audibly indi- 
cate the key via the microphone input or via another type of 
input stimulus. The workstation can analyze the user input, 
determine the key, pitch and tempo for a musical composi- 
tion being partially performed by the user, and adjust and 
transform the composition to be displayed in the new user 
desired key, pitch, tempo, etc., either solely for use on that 
workstation, or communication for use on one or more other 
workstations. In a networked version, this user input can 
also be communicated to other workstations for use by one 
or more of the workstations in transposing, or communi- 
cated to a master workstation, which transposes and rebroad- 
casts the transposed composition. 

Alternatively, the user can input the pitch, tempo, and key 
via the user input (e.g. keypad, joystick, push buttons, voice 
recognition, playing of an instrument, etc.) and the system 
performs the transformation and displays (and/or prints out 
and/or audibly performs) the modified transformed compo- 
sition for the user. Additionally, where a musical composi- 
tion is written for one instrument and a different or addi- 
tional instrument version is desired for simultaneous 
performance, the user can indicate the other instruments via 
the user input, and the system witl generate the appropriate 
displays. The workstation can also provide an audio output 
of the transformed musical composition, either for the 
individual additional instrument or voice transform and 
present it, or for the composite of additional versions and the 
original version, to hear the blended piece. 
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Referring to FIG. 5, a music communication system is 
illustrated comprising multiple workstations (500) each 
comprising a display (510), user input such as a keypad 
(522), a joystick (524), push buttons (525 & 526), a micro- 
5 phone (527), and a speaker (528). The workstation also 
includes communication interface means such as a wireless 
interface including an antenna (531), or alternatively or 
additionally a wired or cabled communication interface 
(540). Each workstation further includes a local microcom- 
w puter subsystem (550) that provides local intelligence and 
management of functions in the workstation. 

In the networked embodiment, where multiple physically 
separate locations each having one or more individual 
workstations provide for communication of performance 
15 data from the individual workstations and presentation by 
the individual workstations of the combined virtual perfor- 
mance data at the individual workstation, to provide for a 
virtual performance. In this case, the communications inter- 
face would utilize slightly different structure, such as a 
20 phone modem (analog modem), a cable modem, ISDN as 
between locations, etc. 

Communications interfaces of various types are well 
known and commercially available. At the present time, they 
are available for purchase at the chip, board, or system level. 
25 In fact, many single chip microcomputers include commu- 
nications interface capabilities, wired or wireless. 

The workstation further includes an optional musical 
instrument input (562) and a musical instrument output 
w (564) that permit the coupling of a musical instrument via a 
musical instrument interface (570) directly to the worksta- 
tion. Thus, a keyboard, electric guitar through appropriate 
input, or a microphone input through the interface (570) 
permits instruments or voices to be directly input to the 
workstation for direct input independent of the microphone 
(527). 

The instrument output permits coupling of the instrument 
input signal, either directly fed through or as modified by the 
workstation for output to the appropriate public address or 

4U amplification and presentation system or separate analysis 
system. The workstations are coupled either via wired or 
wireless communication to a processor subsystem (580) that 
includes a processor, non-volatile memory, read/write 
memory and an interface to a non-volatile storage medium 

45 (582). 

The processor subsystem (580) includes an appropriate 
communications interface, such as a communications inter- 
face (540) for wired interface or (532) for wireless interface 
including antenna (533). The processor subsystem couples 
50 to a non-volatile storage medium (582) containing, among 
other things, application programs, transformation 
programs, and either a shared music library interface appli- 
cation program or the actual shared music library and access 
program. 

55 As described above, the processor subsystem (580) and 
non-volatile storage (582) music library can be built directly 
into one of the music workstations (500) to be a master, with 
the other workstations being slaves, that can either include 
the processor subsystem and non-volatile storage or can be 

60 lower cost dummy slave terminals. As illustrated in FIG. 6, 
a first master workstation (300) provides a basic workstation 
subsystem (200) plus contains the processor subsystem 
(280) and non-volatile storage system (285) as a part thereof 
so as to provide a complete stand alone music communica- 

65 don system, and be capable of acting as a master or 
master/slave. This master workstation^) (300) can function 
as a stand alone, or can couple to one or more other 
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workstations, including one or more masters (300) and/or Alternatively, the microphone input (527) can provide for 

one or more non-master workstations (105). coupling of user speech to a processor sub-system (580) that 

The multiple connected workstations can operate as stand uses any of a number of commercially available and well 

alone workstations using their local intelligence for display- known speech recognition algorithms. These algorithms 

ing downloaded or resident music compositions. They can 5 provide for speech recognition input control, either solely or 

also interact in a master/slave linked environment, where a s a supplement to touch screen or other tactile input 

one of the master workstations (300) asserts a master status, mechanisms. 

and all other inter-connected workstations, whether work- In g de)uxe embodimen[> an outpu t (421) is provided that 

stations (105) or master/slave workstations (300) operate m [{& coupling of an external display, such as a color 

a slave mode coupled to independent on the designated w ^ pT * t J on unit> or other display presentation sys- 

master. Additionally, masters can communicate between . , ,. .. .. ... .... 

each other for a master/master network configuration. tem - cMin S one or more of audl0 > visua1 ' and *f°visual. 

Alternatively, the multiple connected workstations can Additionally, the display presentation output of the work- 
operate together in a networked virtual performance mode, slatl0n «» f ov]de for *> audl ° output presentation of the 
or in a networked communications mode. A dedicated stand „ musical performance, either generated by the work station 
alone or distributed master workstation architecture can 15 responsive to the music composition data, to the users 
provide for the coordination and combination and synchro- performance, to the combined virtual performance data or 
nization of the multiple individual performance data outputs responsive to an external source. Additionally, a visual or 
into a combined virtual performance data output. audl ° vrsual presentation can be provided to provide infor- 

Referring to FIG. 7, an alternate embodiment of the 20 raatl0n fecdback l ? the ™] ™ bolh th f. ir .f™ d ™} 

presen, invention is provided where one or more worksta- 20 mance " wdl as from and between individual workstations 

lions (105) include, at a minimum, a display of the music and/or the master controller or conductor workstation, 

notation. These workstations are coupled to a master music In accordance with another aspect of the present 

communications controller (415) that provides for a separate invention, means are provided for moving through the 

user input (411) which provides input interface, such as to a 2S Pooled (displayed) notation of the music m synchronization 



MIDI status stream, computer data links (such as RS232, 



with the live performance from the displayed musical nota- 



modem data link) etc. that designate requested musical tl0n - 

compositions, transformations, and display requests for vari- Musical notation is used, in the generic sense, to refer to 

ous ones of the coupled workstations. any way of conveying musical performance instructions 

In an alternative embodiment, the master music commu- , 0 inc,udiri g but not ,lmited t0 common musical notation with 

nications controller 415 provides for additional functionality sta ffs, notes, sharps, flats, and clefs, extending to written 

including virtual performance mode, wherein the input inter- instructions in text form to supplement this or supplant or 

face (such as the MIDI stream, computer data links, etc.) partially replace, as well as alternate forms of expression 

provide one or more of musical compositions data for such as chord charts, words and chords (letters), tablature, 

display, transformation information, display requests, user 35 any video, graphic, audio, audiovisual or other display 

individual performance data, and wherein the workstations presentation or combination of the aforementioned types of 

respond to the master music communications controller to presentations. 

couple their individual performance data and receive back An annoyance in performing music using any written 

the combined virtual performance data. notation (whether on paper or displayed on a presentation 

The workstations (105) access the music database storage 40 apparatus such as a screen) is smoothly performing music 

means (420) that provides the data for the requested music and timing "flips of pages" (or the communication of change 

composition via the master controller (415). The master 'o a third party, such as touching a key or button). This is 

controller (415) displays both the requested music compo- especially true when the user is marching and both hands are 

sition as well as user interface communication for the music required simultaneously. 

communication system to be displayed on either a dedicated 45 ln accordance with one aspect of the present invention, 

display (416) or on one of the workstations (105) as desig- means are provided to accept inputs from one or more 

nated by the master controller (415). The music database sources that initiates a "page turn". Types of inputs include 

(420) can either be local, or can be via a data link (e.g., conventional touch input apparatus (such as key switches or 

phone line, RF, otherwise). In one embodiment, a motion capacitive touch pads), motion sensing gear, and automati- 

sensor subsystem (422) monitors motion of a target person 50 cally when operating in the operational mode of Auto Mode, 

and responds in accordance with predefined movement The motion sensing gear can be for a portion of the per- 

interpretation characteristics parameters, such as for a con- former's body, such as a head tilt sensor or an optical eye 

ductor. movement sensor, etc. 

In a preferred embodiment, the user input means (411) is Additional types of inputs that can initiate a "page turn" 

comprised of a key switch device, such as a touch membrane 55 include voice or sound recognition apparatus built into the 

keypad or capacitance touch surface. Alternatively, in one microcontroller system. This apparatus has the ability to use 

preferred embodiment, the user input is provided via a touch pattern recognition specific to the sound or user voice and 

screen technology. Touch screen technology permits the words being said (for extremely high accuracy). Of course, 

display of user interactive icons and legends including text any type of user actuated device such as a foot or hand 

and graphics making possible unlimited customization of 60 switch, or head motion device, or sound or voice recognition 

user input structure according to task needs. Thus, specific system, in a preferred embodiment, is selectively permitted 

switches or sequences of touches to the touch screen can be to control the override of the normal progression of the 

associated with common use icons from the task being music's play. 

performed in conjunction with words to provide ultimate The override may cause the progression to go backwards 

clarity. User error is virtually eliminated, with the aid of 65 or forwards in the music score irrespective of the normal 

automatic entry error detection, such as defined fields, reading of it. The performance mode AutoMode blocks the 

mandatory fields, etc. user override to permit performance according to proper 
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timing and either progresses responsive to the as are well known commercially available including varia- 

music composition data timing, or in an optional tions of analog digital converter input devices, audio signal 

embodiment, to the performer. This automatically moves capture, etc. 

through the musical score as written and preferably shows By combining the conductor and performance mode 

an indication of metronome time and an indication of the 5 operations, the workstation can be enhanced to provide 

proper place in the score where the performer should be for training and comparison of performance to actual music, 

that instrument at any specific time. This is especially Somc music {s omy available in anotatcd forms wherc 

valuable in a conductor mode of networked communication, , here ^ nQt an cxisting signa , showmg proper synchroniza- 

where a conductor couples to one or more music worksta- , ion of the signals Thus> a controller subsystem (such as 

tlons - 10 (580)) provides for real time conversion and analysis of 

The user's performance can be compared to the score, and syntax of the music notation, in conjunction with a precision 

feedback can be provided to the performer as to the quality c i oc k metronome, and provides an indicator (such as color 

of their performance. or other highlighting or holding or accentuating) of the 

In the performance monitor mode, for a single user or relative timing of the performance relative to a place in the 

multiple users, the user (or a remote teacher or conductor) 15 sheet music (or other form of musical notation), 

can indicate the rate at which he feels the performer should Existing forms of music notation can be converted 

be performing. A microphone input on the music worksta- manually, or can be converted automatically by scanning in 

tion samples the user's actual performance and permits sheet music, recognizing (using optical character 

providing a graphical mapping (for the user or teacher/ recognition) the various elements of the music, and facets 

conductor) showing the relative synchronization of the per- 20 arK j specifics of the syntax in the form of notation including 

former's actual performance versus the conductor's desired its constants and variables and protocols, and integrating via 

performance. an artificial intelligence type expert system that notates, 

In an alternate automatic advanced mode, the display of highlights, and accentuates via synchronized metronoming 

the music composition is synchronized to the performers of time signature to music. Any of a variety of other means 

actual performance. Thus, rather than simply indicating "' of converting music can also be used, such as direct input of 

visually for the teacher/conductor or user what their relative musical performance signals processed via software that 

performance was to the written displayed musical converts it into musical notation. Such software is commer- 

composition, the relative performer to written music syn- cially available, such as from ARS NOVA, Wildcat Canyon 

chronization information can be utilized to adjust the display 3() Software, Mark of the Unicorn, Inc., and Passport Designs, 

rate of the actual musical composition to match that of the ' Inc. 

performer. Since the music notation is now in computer usable form, 

With use of appropriate sound baffling, a plurality of it is now an easy task to communicate, display, compose, 

instruments can simultaneously be monitored and controlled alter, and transpose music (such as in key, for types of 

by the conductor, so long as each instrument's output sound 35 instruments or voice parts, and harmonies) via well-known 

pattern is communicated directly to a respective worksta- techniques. 

tion. The output of each of the workstations can then be Additionally, where the user input is converted into user 

coupled to the conductor's master workstation for further performance data for the workstation, the users individual 

analysis and processing. performance data is also now in computer usable form and 

A workstation for an oboe may have a built in slide boom 40 if appropriately converted from performance to performance 
with a featherweight microphone to be able to receive sound data, includes appropriate synchronization data relative to 
input from the oboe. Electric instruments, such as guitars, master controller performance synchronization signal. Thus, 
keyboards, and other electrical analog signal sources can be both the music notation is in computer usable form (making 
fed directly to a line input that is appropriately buffered and it easy to display, alter and analyze/compare), and the users 
filtered. Signal input can also be accommodated through a 45 performance is in computer usable form (digital individual 
MDI-interface sub-system that permits both utilization of performance data), it is possible to provide intelligent opera- 
data in workstation to workstation communications and tion and analysis and utilization of both the music compo- 
utilization of MIDI-output at the station where the data was sition information and the user performance information, to 
input. provide for various automated modes and features to the 

For networked virtual performance, and for one aspect of 50 users Implementation can also be in a custom design corn- 
output display presentation, the utilization of MIDI input prised of a microprocessor, non-volatile storage memory, 
and MIDI output, at each of the individual workstations and read/write memory, and whatever additional peripheral cir- 
at the master controller workstation, permits the capture or cuitry is needed (such as are available in ASICs, or single 
the user performance and conversion to individual perfor- chip micro-computer chip sets including CPUs, DSPs, A/D, 
mance data which includes time synchronization informa- 55 and other peripheral support circuitry). These single or 
tion which can be communicated to the master workstation, multiple chip solutions can be utilized to create a dedicated 
which synchronizes and combines the individual perfor- system to perform complete music workstations perfor- 
mance data to generate combined virtual performance data mance criteria to support an extremely low cost, high 
which is then communicated back to the individual work- volume music workstation solution, 
stations which utilizing their MIDI output interfaces provide 60 A new form of communication is created in that both the 
for display presentation (e.g. audio output) of the combined process of communicating via standard notation is respected 
virtual performance data. Additionally, even where virtual and adhered to, while at the same time permitting interaction 
performance mode is not selected, the provision of MIDI and communication of music media signals ranging from 
interface input and output on the workstations have other simple analog, digitized (analog to digital converted), indi- 
multiple beneficial users as discussed elsewhere herein. 65 vidual performance data (representative of the user's 
Additionally, other types of user performance to user per- performance), and can optionally include timing/ 
formance data input devices and transducers can be utilized, synchronization data. 



6,084,168 

21 22 

Amulti CD ROM changer accommodates indexed storage orchestra section is to reduce volume. In this mariner, either 
of hundreds of thousands to millions of musical composi- camera input of movements, glove sensing of movements, or 
tions to permit complete stand alone operation of the user other techniques (such as audio, ultrasonic, etc.) can be used 
music workstation. Alternatively, an optional built-in or to track movement to permit associated meanings to be 
external modem can be provided to permit inter- 5 attached or indexed to particular signal parameters or para- 
communication with a remote central music database man- melric si 8 nals of the meaning of the movement parameters 
agement system that permits both communication and down as provided by the conductor input device. For example, in 
loading (and disconnect) for stand alone operation. Thus the the case of the ~' ^ g love ; that mputwould be the 
workstation can stay on-line, pulling up music as needed, or S1 !f al out F ut of the as interpreted by associated 
can request a single or multiple pieces of musical works be 10 twar * m * Processor (such as a PC or a MAC), 
provided to it, that are then down-loaded from the central Alternatively for example, in the case of video camera 
database manager. The user workstation then disconnects 1D P ut - 11 could , be pattern recognition or analog s.gna 
from the music database management system, and thereafter comparison lo determine the presence of certain signal 
operates stand alone where all desired music is stored locally P al,erns '"dicating to the system to initiate automatic com- 



in storage (preferably non-volatile). Storage can be 15 
semiconductor, magnetic, optical or any other medium. 



munication of a conductor presentation. In so doing, the 
conductor is able to rapidly convey his meaning, focus it to 



_ particular group of instruments, and be done with it. He 

The use of virtual reality technology, including motion doesn>1 have [0 focus very long or concentrate t0 make sure 

sensors and body gloves, permits monitoring ot various m , ye n his signa , Inst£ad he can focus Qn listening 

other things (as shown in FIG. 9). For example, as shown in (Q see ;f m his message 

FIG. 10, a camera in conjunction with analysis logic, such 20 ^ g ^ embodiment of lhe present 

as expert software, can monitor motion of role model invenlion _ In this embodim ent, the workstations are remote 

behavior and compare performer behavior. Hand, finger, ^ m _ m) used b a fflember of a marchm band Each 

arm, leg, eye, head, body, and mouth movements can all be rf ^ Kmo{c ^ (801 _ 803) are ; d with reC e ivers 

monitored and constructive critical feedback can be (810 ^ 12) that receive musical compositions transmitted to 

accumulated, analyzed, and fed back to the user or teacher, 25 ^ ^ (820 _ 822) contro , , he 

for performer training, or performances, or for conductor tjon rf ^ remote nmt ^ musica] composition 

communication. fa displayed on , h c remote unit's displays (830-832) which 

The input of monitored movement data is provided to the displays can be an LCD multiple line display providing low 

user workstation, permitting precise mechanics training such C ost, low pow er usage, and high visibility /readability, and 

as finger position, the angle of striking of strings relative to with Aut0 Advance Mode, the display automatically scrolls 

the neck of a violin or guitar, or they can be used to permit as tne music ^ to be per f orme d. 

the virtual performance of music by a performer using a Each remote um , can be mounted on the 

virtual link apparatus such as a virtual reality glove and head irjstrumerjt on or in place of the lyre. The remote unit's 

movement detection apparatus. The user can then perform a anteMa (84 o_ 842 ) can be separate from or built into the 

piece with their own personalization without any musical remo(e uni( Qf , he lyre 

instrument in fact. A lransportab]e main urjit ( 850 ) ; s used to transmit musi- 
For example, the guitar portion for a piece of music could cal compos jtjons to the remote units (801-803). The trans- 
be displayed in notation form and actually performed port able main unit (850) is comprised of a controller (806) 
according to the timing of movements of the user's fingers 4(J for controlling the transportable main unit (850), a music 
(cither actual fret positions, or only timing information). To database storage medium (805) containing the data for the 
add further reality, a mock guitar, keyboard, flute, or other mus j C al compositions to be played by the band, and a 
instrument can be used and combined with virtual effects to transmitter (804) for transmitting the musical compositions 
provide for music performance and personalization. Thus, to , he remo t e units (801-803). This main unit can be in the 
for entertainment purposes, users could perform as part of a 4 , form of a sm ,c ase or briefcase size item. The main unit can 
symphony orchestra playing a violin portion. If they per- also be provide d built into a van that is driven around with 
formed out of time, they would hear their instrument's lhe ban d 0 r as a small self-contained portable unit. In 
performance out of synch with the rest of the orchestra's accordance with this embodiment, the band can play a 
performance. virtually unlimited number of musical compositions without 
There arc numerous ways to embody the conductor move- 50 the problem of carrying the music with them in paper form, 
ment interpretation system. As illustrated in FIGS. 9 and 10, It also relieves the band members of the problems of 
one is utilizing the body movement detection apparatus changing music and changing pages while marching. As 
prevalent in virtual reality, sports medicine, etc., as dis- discussed in the above embodiments, in the performance 
cussed above, to identify specific movement patterns or mode, the musical score is automatically scrolled across the 
signal parameters associated with certain movement 55 screen display (830-832). Additionally, a keyboard and/or 
patterns, to initiate a display presentation, audio, video, or microphone can be attached to the transportable main unit 
audiovisual to provide a presentation associated with move- allowing the conductor to send messages to the remote units 
ment of the conductor. Alternatively, other techniques can be via displays (830-832) or via a speaker associated with units 
used such as taking the video feed from a video camera or (801-803). This allows the conductor to send instructions to 
other video source (e.g. VCR) and having the conductor 60 the band (such as to take a certain route, or play at different 
interpret his movements and assign them unique meanings, volumes or speeds). With bidirectional communications and 
to create a lexicon of his movements and corresponding user performance feedback, the conductor can also monitor 
meaning. for errors. 

For example, rapid downward movements of the hand FIG. 9 illustrates a conductor, stage hand, or other person 

from up to down, in a certain manner, indicate "decrease the 65 with a sensor glove on each hand (935) and a head and eye 

volume." When he points at a particular section at the same movement monitor (930). The figure also illustrates the 

time as he is doing that, he is indicating that only that conductor wearing Ml body sensor equipment (940). Either 



virtually linked to perform), with or without a blue screen 
(1010) behind him. The video camera (1005) feeds a video 
signal to the video processing system (1020) that utilizes 
signal processing to provide signal pattern recognition capa- 
bility. The blue in the screen is filtered out in the signal 
processing such as by an Ultimatte process. 
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embodiment or a combined embodiment can be used to map is made whether the conductor automated mode is selected, 

body movements. If only the gloves (935) or body sensors and if a "yes" decision, then automated Mode 4 is entered at 

(944) are used, the movement of the glove or sensors can be Step 1149, and if a "no" decision, then logic goes to the 

captured by a video system, as illustrated in FIG. 10. networked virtual performance mode A Mode 5, as decided 

Other methods that capture motion rely on specialized 5 at decision logic step 1150, and responsive to the selection 

sensors (944) placed on a performer's joints, such as via a * networked virtual performance mode, A Mode 5 

sensor body suit (940). Once motion has been fumed or * If networked mode A Mode 5 is not 

analyzed, a data set is produced to interpret that movement selected then processing proceeds to decision logic step 

into Cartesian co-ordinates. These co-ordinates provide the l } 52 to f dec,de whether the synchronization of the display to 

spatial location of each of those markers. This information 10 * e Performance (Mode A 6) has been selected and if so, 

is then cleaned up and input to an animation package. the processing proceeds at step 1153 to provide A Mode 

„,„ .„ .„ , * j . a- 6 functions. The automated mode processing ends at step 

FIG. 10 illustrates a video camera (1005) and a standing m4 |( ^ ^ ^ ^ ^ ^ automated modes can be 

u^ttZ cumulatively selected, so that the networked virtual perfor- 
mance can include any or all of the other automated modes, 
including A Modes 1, 2, 3, 4, 5, and 6, or other A Modes 
which may be provided. It should also be noted that the 
tracking of the performance by the user relative to the 
presentation of the musical composition (step 272 of FIG. 
2D) provides the first part of the logic necessary for the A 
In one embodiment, the conductor is wearing a sensor ~" Mo(k 6 (synchronization of dis pl ay t0 performance), as 
equipped body suit (940) and gloves (935) of FIG. 9. In illustrated m FIG. 13 in further detail, 
another embodiment, the conductor is wearing only the Referring t0 FIG . 11B there is illustrated an alternative 
sensor equipped gloves (935) of FIG. 9. In still another overalJ ^ rf & m ^ c com itlon communications 
embodiment, the conductors movements are picked up by ^ alternative to that illustrated in FIG. 2A, showing 
the v,deo camera (1005) and processed without a sensor suit. - addilional au , om ated modes including the network virtual 
Simple things, like looking for the conductor's rapid hand performance mode A 5. FIG. 11B illustrates an alternative 
movements, focusing on specific hand movement areas, control logic structure for the network selection, where the 
facial and head movement, arm movements, and body decision logic proceeds in an analogous manner to FIG. 2A. 
language can all be programmed into the recognition knowl- , 0 xhe opera ti 0 n starts at step 1100, and a mode is selected at 
edge base. Some of the technology lor complete mapping of step mo A compos ition is selected at 1120, and if the 
body movement that exists in making video games of today decision logic 1120 determines the composition has been 
are illustrated in Video Systems magazine, page 42, October selected, then processing proceeds at step 1130, which 
1995, Vol. 21, No. 11, and NEXT Generation magazine, prov i des for selection of the mode. If no composition has 
pages 49-54, October 1995, both incorporated herein by 35 been selected, processing proceeds back to the select corn- 
reference, position step 1110. Once the composition has been selected, 
In any event, having now obtained knowledge related to then a Mode is selected as illustrated at step 1130, and the 
recognition of the movements, the system can interpret them decision logic of select Mode 1140 determines whether the 
and utilize them to convey presentation information to the selected mode is that of (1) a display mode, in which case 
ensemble or orchestra or studio members, or to analyze a 40 processing proceeds to step 220 of FIG. 2A, (2) a modified 
performer's movements, or to permit a virtual performance. composition mode, in which case processing proceeds as 
One example would be a large screen television or multiple from step 215 of FIG. 2A going to the yes branch, or (3) the 
large screen televisions for viewing by the members of the select mode determines that the networked virtual perfor- 
viewing group. Alternatively, each music stand could pro- mance mode has been selected, in which case processing 
vide for a picture in picture display of special movements of 45 proceeds to the networked mode automated Mode 5, step 
the conductor in areas of the display where music is not 1150, as illustrated in further detail in FIG. 12. 
currently being played. Since the stand can have the intel- Referring to FIG. 12, there is illustrated the networked 
ligence to compare the performed music to the played music, virtual performance process flow, step 1150 of FIG. 11B, in 
that embodiment permits display of the message in portions greater detail. At step 1150, the automated mode of net- 
of the music display area which have already been per- 50 wor k e d virtual performance has been selected. This auto- 
formed or arc not going to be performed for some time (e.g., mated mo de 5 is processed by proceeding to verify the 
at least ten seconds in either direction; other criteria could selection of the composition. At step 1220, the user's 
alternatively be used, and can be set up for desired charac- performance is captured (e.g., by a microphone or via MDI) 
teristics of the performing environment). an d converted to individual workstation performance data. 

Voice recognition and response to conductor commentary 55 This data is then broadcast by each of the plurality of 

can supplement the system. The system could record the individual music stands to a designated master controller. At 

message, interpret to whom the conductor directed the step 1230, the controller receives and combines the plurality 

message and convey it audibly or translate it into a text or user's individual performance data to generate a combined 

icon display as a part of the system's audiovisual presenta- performance data representative of the synchronized and 

tion. 60 combined plurality of individual performance data This 

Referring to FIG. 11A, an alternative embodiment of the combined performance data is then broadcast to all of the 

automated mode "A Mode" (240) of FIG. 2C is illustrated. individual music workstations. At step 1240, each of the 

The illustrated process for the automated Mode 240 of FIG. plurality of individual music workstations provides an out- 

UA includes process steps for the Auto Advance A Mode 1, put (e.g., audio) responsive to the combined performance 

Training A Mode 2, Performance Mode A Mode 3 which are 65 data, which is representative of the combination of the 

equivalenttotheillustratedAModesl,2,and3ofFIG.2C. plurality of users audio performances of the music 

However, as shown in FIG. 11A, at step 1148, the decision composition, synchronized to the display presentation of the 
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music composition. Timing synchronization data is provided 
to synchronize the plurality of individual music workstations 
individual performance data to one another and to the 
selected musical composition's display presentation, to 
facilitate synchronization for purposes of combination and 
audio playback at the individual workstations, in approxi- 
mately real time, so as to provide apparent real time feed- 
back of the individual user's performance in combination 
with all other individual users, played back at each of the 
individual workstations. At step 1250, analysis is done as to 
whether the composition is done. If so, processing completes 
at step 1260 and ends this automated Mode 5. Otherwise, if 
processing is not done, that is the musical composition is not 
over, then processing proceeds back to step 1220 as illus- 
trated. 

Referring to FIG. 13, there is illustrated automated Mode 
6 corresponding to synchronizing the actual display presen- 
tation to the user's performance, providing resynchroniza- 
tion of the display of the music to account for the actual 
performer's timing, and any errors or differences. Auto- 
mated Mode 6 is analogous to A Mode 1 without override, 
but not to A Mode 3, which auto-advances independent of 
the user's performance. At step 1153, the automated modes 
6 is entered. At step 1310, analogous to step 272 in FIG. 2D, 
the user's performance is tracked and stored for comparison 
to the musical composition display presentation relative to 
the timing of where the performance should be. At step 
1320, the performance that has been tracked is compared to 
the score and deviation performance data for that performer 
is generated. At step 1330, the errors are utilized to resyn- 
chronize the display presentation of the musical composition 
to adjust to the actual user's real time performance timing 
relative to the display presentation and a synchronization of 
the commencement of the user's performance relative to the 
musical composition's presentation on the display. At step 
1340, an analysis is done as to whether processing is 
complete (e.g. is the musical composition done?). If so, A 
Mode 6 ends at processing step 1350, and either the auto- 
mated modes of processing can end, or other automated 
modes can be checked for, and processing can begin for 
another selected automated mode. 

Referring to FIG. 14, there is illustrated the communica- 
tions architecture and system in accordance with the present 
invention, wherein there are a plurality of individual music 
workstations 1410, each responsive to a user's performance 
of a musical input (1411) which is processed by the indi- 
vidual music workstation 1410 to provide a performance 
data output 1412 coupled for communication to a control 
system 1450, which can be a master controller or master 
workstation, which is coupled for receiving and processing 
the individual workstation performance data for the plurality 
of individual workstations. A communications interface 
1413, such as a modem, can provide for coupling and 
communication between the individual workstation 1410 
and the master controller 1450. In one embodiment, the 
modems 1413 are coupled via a public switched telephone 
network to a corresponding modem 1455 which couples to 
the master workstation to communicate by directionally 
between the individual workstations and the master work- 
station. Alternatively, any sort of cable, hardwired, or wire- 
less communications interface and provided, and would 
work equally well when in accordance with the present 
invention. The master controller 1450 is responsive to the 
individual performance data outputs from the plurality of 
individual workstations to provide a combined and synchro- 
nized virtual performance data output 1451 which is coupled 
via the communications network to the plurality of indi- 
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vidual workstations, which each then respectively provide 
an output (e.g. audio) 1415 for presentation to the user, 
representative of the combined performance data. An audio 
and/or visual presentation output 1452 can also be provided 
at the master workstation. 

Referring to FIG. 15, there is illustrated the data flow and 
communication architecture for one embodiment of the 
present invention. A music communication system is pro- 
vided that has a plurality of physically separate remotely 

10 located locations, each having one or more individual music 
workstations, wherein each of the individual music work- 
stations provides for communication output of individual 
performance data for a user's performance of a displayed 
musical composition at that individual workstation. As 

15 illustrated, a master controller 1505 is coupled to the com- 
munications infrastructure 1510 at a first location. A plural- 
ity of individual music workstations 1511, 1512, 1513, and 
1514 are coupled to the communications infrastructure 1510 
at location 1, designated performance group 1. Similarly, 

20 location 2, performance group 2, provides individual work- 
stations 1521 and 1522 coupled to the communications 
infrastructure 1510, and location 3, performance 3, provides 
individual music workstation 1531, 1532, and 1533 coupled 
to the communications infrastructure 1510. The plurality of 

25 individual workstations 1511, 1514, 1421-1522 and 
1531-1533, each provide individual performance data, pro- 
viding multiple discrete time samples of performance data 
(e.g. time, segments of data), which can include synchroni- 
zation information therein, which are communicated via the 

30 communication infrastructure 1510 to the controller or mas- 
ter workstation 1505 The resultant output (e.g. audio) from 
the individual workstations is the resynchronized combined 
performance data of the plurality of individual performance 
data, resynchronized for each of the time segments of 

35 output. 

Referring to FIG. 16, the data word structure for one 
embodiment of the present invention is illustrated. The data 
structure for the individual user performance data is illus- 
trated as comprising a header comprised of control infor- 

40 mation including a start of word, cyclic redundancy code 
(CRC) word, the performer's ID (a unique address for the 
individual workstation, (which not only can provide an 
unique ID but can also provide for indication of an instru- 
ment type), and other control information for other modes of 

45 operation in addition to the virtual performance), payload 
data representing the actual performance data, and timing 
synchronization data. Any other data word structure provid- 
ing for header information including control information, 
and payload information including performance data, and 

50 some timing synchronization scheme, whether or not part of 
the user performance data word will work equally well with 
the present invention. 

FIG. 17 illustrates a networked system embodiment in 
block diagram form, illustrating the network interface sub- 

55 system and music data processor subsystems for one 
embodiment of the present invention, compatible with the 
architecture as described in FIGS. 1-15. FIG. 17 illustrates 
a single individual workstation 1710, although a plurality of 
individual workstations 1710 are usually involved in the 

60 networked system (analogous to the plurality of individual 
workstations 1410 of FIG. 14). The individual workstation 
1710 provides for communication with a master workstation 
1790, (analogous to the communication architecture with the 
master workstation 1450 of FIG. 14). The musical perfor- 

65 mance by the user (1712) is coupled to an input interface 
1715 which provides musical performance data to a musical 
data processor 1720, which can be a MIDI based processor, 
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which provides the music processed performance data out- 
put to a network communications interface 1730 which 
outputs the individual musical performance data 1731 for 
coupling via the communications infrastructure to the master 
workstation 1790. 5 

The master workstation 1790 processes the individual 
musical performance data 1731, from the plurality of indi- 
vidual workstations 1710, and provides an output 1791 of 
combined virtual performance data, wherein both the indi- 
vidual musical performance data and the combined virtual 10 
performance data are comprised of a plurality of discrete 
time segmented data samples, wherein the combined virtual 
performance data is synchronized within each sample time 
to provide for synchronized combination of the individual 
musical performance for that time segment 15 

A network interface subsystem 1735 is responsive to the 
combined virtual performance data to provide synchronized 
virtual performance data output to the music data processor 
1725 (e.g. MIDI) which provides a sound output of the 
combined virtual performance data corresponding to the 20 
synchronized combination of the individual musical perfor- 
mance data from a plurality of the individual workstations 
1710. Additionally, the individual workstation 1710 pro- 
vides data analysis and presentation processing 1740 which 
provides an output responsive to, inter alia, the combined 25 
virtual performance data to a presentation means such as an 
audiovisual display, or a video display or an audio output, 
1745. Note that in a preferred embodiment, the multiple 
individual network workstations 1710 are synchronized (e.g. 
responsive to synchronization signal from the master work- .in 
station 1790), so that the plurality of the individual work- 
stations simultaneously commence display presentation of 
the musical composition for synchronization of individual 
user performances. In one embodiment, the output of the 
individual performance data (such as via a MIDI music 35 
processor) is comprised of timing synchronization informa- 
tion that is used by the master workstation to synchronize all 
of the musical performances representative individual musi- 
cal performance data to virtually perform together in a 
combined and time-synchronized performance. The com- 40 
bined virtual performance data representing the synchro- 
nized virtual performance is then sent back (e.g. broadcast) 
to the individual workstations, coupled via the network 
communication interface 1735 back to the individual work- 
station's music data processor 1725, to provide audio and/or 45 
visual output corresponding thereto. Where the master 
workstation is for a conductor or teacher, the conductor 
mode or teacher mode can also be utilized in addition to the 
network virtual performance mod. In fact, all of the auto- 
mated modes can be selectively combined, such as by 50 
providing layering of functions via software. 

Referring to FIG. 18, the timing diagrams for the indi- 
vidual performance data and synchronized virtual perfor- 
mance data is illustrated, relative to signals output from the 
individual networked workstations NWSj, NWS 2 , NWS^ 55 
and a master workstation MWS. As illustrated, the master 
workstation MWS provides a synch signal to commence 
synchronization of all workstations (NWS) coupled to the 
system. In a preferred embodiment, the individual worksta- 
tions (NWS) begin the display presentation of the musical 60 
composition (as does the master workstation) in synchroni- 
/.alion with the synch signal from the master workstation. 
Responsive thereto, each of the individual workstations 
displays its musical composition, and the users of the 
workstations can perform the musical composition. Indi- 65 
vidual workstations (NWS) provide individual performance 
data responsive to the user's performance. As illustrated, 
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workstation NWSj provides performance data A and B at 
respective times, while in the individual workstation NWS 2 
provides individual performance data C & D at respective 
times, and independent workstation NWS W provides indi- 
vidual performance data signals E & F at respective times. 
During the first time segment, individual performance data 
A C & E are output, respectively, from individual work- 
stations 1, 2, and N. However, as shown in FIG. 18, the 
relative timing of output of the individual performance data 
(and receipt by the master workstation thereof) are skewed 
from one another in timing. The master workstation resyn- 
chronizes the individual performance data A, C, and E 
within each time segment, and provides an output of the 
recombined reconstructed and synchronized A, C, and E to 
provide synchronized combined virtual performance data. It 
is to be noted that synchronized virtual performance data A, 
C, E occurs within the first time segment Tl, which contains 
both the individual performance data and the corresponding 
responsive synchronized virtual performance data. The 
maximum time delay T^, represents the acceptable worst 
case time that a user can tolerate between the user's perfor- 
mance output of corresponding individual performance data 
and the time until the synchronized virtual performance data 
corresponding to that individual performance data is 
received by the same individual workstation and provided as 
audio output. Similarly, during time slot 2, each of the 
individual workstations provides individual performance 
data B, D and F, slightly time skewed from each other, and 
the master workstation provides combined synchronized 
virtual performance data B, D, F, which is communicated 
back to the individual workstations to generate an audio 
output. 

The present invention finds application in many different 
environments and embodiments. For example, a live prac- 
tice can be held from multiple locations, while one or more 
musicians are editing the score and communicating the 
changes real time, repcrforming them, etc. An orchestra can 
be at one location, while a singer can be at another location. 
Creating and editing and synchronizing music to movies 
scores can also be facilitated with the present invention with 
beneficial gains. For example, in a movie, a lot of elements 
are changing dynamically that need to be resynchronized. 
For example, a scene can be lengthened or shortened, and 
the soundtrack must be adjusted. A score which was created 
to the movie must be rewritten to accommodate the edited 
movie. Typically, the composer creates the score at home or 
at a studio. That score is sent to the movie studio. However, 
the movie studio may edit the scene that that music is used 
in, say a reduction of three seconds in the scene. The edited 
scene is sent back to the composer so that he can shorten his 
music. The process is slow, inefficient, and an impediment to 
the production process. By utilizing the present invention, 
this can be accommodated in real time, and from remote 
locations, and the virtual performance mode as well as the 
other modes of the music communication of the present 
system can be utilized advantageously for this application. 

The present system provides an infrastructure for a music 
Internet. It can be applied to interactive editing of music. It 
can provide for special one on one mode for teaching and 
performance evaluation. It can facilitate adapting a music 
display to the performer's performance, or drive the display 
to a metronome and force the user's performance to be 
synched to the metronome timing. Music can be composed, 
edited, transposed, etc. easily, from any of the plurality of 
individual workstations or on a single standalone worksta- 
tion. Multiple individual music workstations (INW's) can 
communicate with one another, both with data and audio 
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such as voice and/or music, and/or supplemental data such 
as video, audio, data files, control signals, applets, etc. 
Bands can practice at home without the disadvantage of 
either practicing alone or having to physically get together 
in a large grouping Singers can group with other instruments 
and other vocalists via respective ones of a plurality of 
IMW's, one, two, or even the whole plurality being remotely 
located physically from one another (e.g., LA, NY). 

'iliere can be multiple instruments playing at a single 
location. There are many options regarding how IMW's are 
allocatable, such as. (1) on IMW (Individual Music 
Workstation) per location, (2) IMW per instrument type, (3) 
one IMW per performer (e.g. Instrument, vocal). There are 
also many options (1) for networked communications 
between the IMW's and/or between the IMW's and the 
Master Workstation. Modes can be selectively mixed and 
matched and combined, such as (1) all auto-modes enabled 
mode (2) virtual performance mode, full audio-visual, (3) 
audio link only mode, etc. There are also many options as to 
modes of signal processing which can be applied to the user 
input to generate the individual performance data, ranging 
from simple analog to digital (A/D), to complex musical 
sound analysis algorithms (e.g. to detect characteristics such 
as instrument type, pitch tone, attack, decay, amplitude, 
voice signatures, etc.) There are also tremendous numbers/ 
parameters of performance analysis algorithms that can be 
selected and which utilize the selected musical composition 
data (corresponding to that displayed on the IMW which is 
receiving the performance input for analysis) (e.g. the sound 
characteristics generated by the sound analysis of the user's 
performance, to (3) in between "(1)" and "(2)". Additionally, 
many kinds of data can be communicated between (1) 
IMW's (2) IMW's and a Master Workstation, and (3) IMW's 
and external communication interfaces, etc. Examples of the 
kinds of data include performance data, video data, 
processed/analyzed performance data, audio signals, data 
files, etc. 

Referring to FIG. 19, the memory work space structure 
for both the individual and the master workstations, in the 
virtual performance mode, is illustrated. Each of the indi- 
vidual workstations provides for music data buffering o the 
outgoing live performance data, individual performance data 
IPDXXX, and further provides for storage of the incoming 
combined virtual performance data CPDXXX. This buffer- 
ing is done within read-write memory within the individual 
IMW's. Similarly, the data structure for the memory buffer 
of the master workstation is illustrated, showing separate 
buffer storage locations for each of the plurality of individual 
workstations WSj to WS^, providing storage of associated 
respective individual performance data IPD A-N. In one 
embodiment, multiple frame buffers are provided in the 
IMS, so that multiple time segments of IPD and CPD data 
can be stored in the buffer structure, and so that the buffer 
structure can support both input storage and output retrieval 
operations. Other data structures and buffering operations 
are also compatible with the present invention. The structure 
of the data word can be changed or adapted within its 
existing structure to facilitate the transfer of supplemental 
data is transferred from the IMS or Master Workstation, or 
between the IMW's, or between the IMS's and the Master 
Workstation. By utilizing the illustrated or other compatible 
data buffer structure, timing and synchronized (with the 
frame buffered variety) asynchronous communication are 
facilitated, not only as between communication 
workstations, but furthermore, as to the operations internally 
of the IMW. With frame buffering, the individual worksta- 
tion and master workstation can dump the result output to 
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the buffer, or receive input individual performance data 
asynchronously and buffered, relative to the other operations 
of the respective workstation, The IMW can facilitate opera- 
tion in one mode or in multiple modes concurrently. 

5 Prom the foregoing, it will be observed that numerous 
variations and modifications may be effected without depart- 
ing from the spirit and scope of the invention. It is to be 
understood that no limitation with respect to the specific 
apparatus illustrated herein is intended or should be inferred. 

10 It is, of course, intended to cover by the appended claims all 
such modifications as fall within the scope of the claims. 
What is claimed is: 

1. A music workstation system for use by a user in 
providing a performance of a displayed video display pre- 

15 sentation of a selected musical composition, said system 
comprising: 

storage means for storing composition data representative 
of the selected musical composition; 
, () display means for displaying the video display presenta- 
tion responsive to the composition data; 

input means, responsive to the performance by the user, 
for providing user performance data output; 

analysis means for comparing the user performance data 
25 to the respective associated composition data; and 

control means for modifying the display of the composi- 
tion data responsive to the analysis means. 

2. The system as in claim 1, wherein the modifying is 
comprised of resynchronizing the fixed rate to resynchroni/.e 

30 the display of the composition data to the performance by 
the user responsive to the analysis means. 

3. The system as in claim 1, wherein the modifying is 
comprised of displaying performance variance data respon- 
sive to the analysis means. 

35 4. The system as in claim 1, wherein the workstation 
system is further comprised of means for modifying the 
sequencing and timing of the display presentation respon- 
sive to input from the users. 

5. The system as in claim 4 wherein the composition data 
40 is comprised of timing information data and wherein the 

sequencing and timing of the display presentation is con- 
trolled responsive to the timing information data. 

6. The music workstation system as in claim 1, wherein 
the 

45 control means provides for a display presentation of the 
differences between the displayed composition data 
and the performance data, responsive to the analysis 
means. 

7. The system as in claim 1 further comprising: 
processor means for generating a video display output 

responsive to the composition data in the storage means 

for the selected musical composition; 
wherein the display means provides the video display 
55 presentation responsive to the video display output; 
wherein the input means is further comprised of storage 

for storing user performance data concurrent with the 

corresponding display presentation; and 
wherein the control means modifies the video display 
60 presentation to reflect the result of the comparing the 

user performance, data, and the composition data. 

8. The system as in claim 1, wherein the user performance 
data is comprised of at least one of audible performance 
data, visual performance data, and control data. 

65 9. A music display presentation system for use by a user 
in viewing a selected musical composition, said system 
comprising: 
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a workstation apparatus, comprising: 

storage means for storing musical composition data rep- 
resentative of musical notation and display presentation 
corresponding thereto; 

display means for providing a display presentation s 
responsive to the musical composition data; 

means for progressing the display presentation responsive 
to the musical composition data; 

mode controller means for managing the operation of the 
workstation in at least one selected automated opera- 10 
tional mode to provide for automated progression of the 
display presentation of the musical composition; 

wherein the selected automatic operational mode is com- 
prised of at least one of marching band mode, conduc- 
tor mode, orchestra mode and networked virtual per- 15 
formance mode. 

10. The system as in claim 9 further comprising: 
input means, for providing an output of user performance 

data responsive to performance by the user 
analysis means for analyzing at least one of the user 20 

performance data, the musical composition data, and 
control means for providing an output of individual 

workstation data responsive to the analysis means. 

11. The system as in claim 9 wherein the musical com- 
position data is further comprised of temporal data, 2 5 

wherein the display presentation is logically sequenced 
responsive to the temporal data. 

12. The system as in claim 9, wherein the workstation 
provides user instrument type data, wherein the mode con- 
troller manages selection of the transposition responsive to xi 
the user instrument type data responsive to the user instru- 
ment type data. 

13. The system as in claim 12 wherein the user instrument 
type data is determined responsive to at least one of a live 
performance by the user, a performance by the user and a 35 
user input selection. 

14. The system as in claim 12, wherein the user instru- 
ment type data is representative of at least one of voice, 
musical instrument by type and electronically generated. 

15. The system as in claim 9, wherein the workstation is 40 
operable in a plurality of operational modes concurrently. 

16. The system as in claim 9 wherein the workstation is 
coupled for communication with at least one other one of the 
workstations to form a network of workstations. 

17. The system as in claim 16 wherein one of the 45 
workstations in the network functions as a master controller 
and collects and stores the individual workstation perfor- 
mance data from the plurality of workstations coupled in the 
network. 

18. The system as in claim 17 wherein the master con- 50 
trailer processes the stored individual workstation perfor- 
mance data to provide an output of combined stored per- 
formance data to all the workstations. 

19. The system as in claim 18, wherein the plurality of 
workstations in the network are synchronized to each com- 55 
mence the display presentation at approximately the same 
time. 

20. The system as in claim 19, wherein the individual 
workstation performance data for the plurality of worksta- 
tion are synchronized and combined to provide the com- 60 
bincd performance data. 

21. A virtual music performance system for use by a 
plurality of users in providing a plurality of display presen- 
tations of a selected musical composition, said system 



: workstation (IMW), each 



i plurality of individual 1 
IMW comprising 



a communication interface providing for communica- 
tions of musical composition data of the selected 
musical composition with the respective IMW; 
a computing subsystem providing processing and 
memory for locally storing the musical composition 
data responsive to the communications interface; and 
a music display apparatus for providing a local visual 
display presentation of the selected musical compo- 
sition responsive to the computing subsystem; 
wherein the system further comprises means for control- 
ling a plurality of the IMWs to concurrently provide a 
plurality of local display presentations of the selected 
musical composition. 

22. The system as in claim 21, further comprising: 

an input device responsive to a performance by the user 
of the displayed musical composition for providing an 
output of user performance data. 

23. The system as in claim 22, wherein the computing 
subsystem provides processing means responsive to said 
user performance data for providing an output of individual 
performance data out. 

24. A musical data communication system enables a 
presentation of a selected musical composition for use by a 
plurality of users, said system comprising: 

a plurality of individual workstations each providing 
means for independently capturing the musical perfor- 
mance of a respective user and communicating indi- 
vidual performance data; 

control means for synchronizing and combining the indi- 
vidual performance data from the individual worksta- 
tions to provide an output of combined performance 
data; 

means for communicating said combined performance 
data to the plurality of individual workstations; 

wherein at least one of the individual workstations pro- 
vide at least one of an audio, video, and audiovisual 
output representative of the combined musical perfor- 
mances for all of the communicating plurality of indi- 
vidual workstations. 

25. The system as in claim 24 wherein the plurality of 
individual workstations provide for synchronized display 
presentation. 

26. The system as in claim 25, wherein each of the 
individual workstations provide for output of individual 
performance data representative of the musical performance 
by the user corresponding to the display presentation. 

27. The system as in claim 25 wherein the individual 
performance data is further comprised of timing synchroni- 
zation data. 

28. The system as in claim 26 wherein the control means 
provides a synchronization signal, wherein the individual 
workstations provide said timing synchronization data 
responsive to the synchronization signal. 

29. The system as in claim 24 wherein each of the 
individual workstations is further comprised of a network 
interface subsystem and a music data processor. 

30. The system as in claim 24, wherein a plurality of the 
individual workstations provide at least one of an audio, 
video, and audiovisual output representative of the com- 
bined musical performance. 

31. A musical display system comprising: 

a memory, for selectively storing musical composition 
data representative of a selected musical composition; 

means for producing a display presentation responsive to 
the respective musical composition data; 

selection means for determining a selected display format; 
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data processing means responsive to the selection means 
for selectively processing the stored musical composi- 
tion data responsive to the selected display format to 
produce a corresponding particular display presentation 
format responsive to the musical composition data; and 5 

at least one display subsystem, responsive to the selected 
music composition data and the selected display 
format, for displaying a video presentation of the 
musical composition in the selected display format. 

32. The system as in claim 31, wherein said processing 1Q 
comprises at least one of transposing and communicating 
external to the musical display system. 

33. The system as in claim 29, further comprising: 
operational selection means for determining a selecting 

operating mode wherein the data processing means is ^ 
responsive to the selected operating mode for control- 
ling progression of the video presentation. 

34. The system as in claim 31, wherein responsive to the 
selection of the display format, the data processing means 
provides for additionally displaying at least one of user ^ 
fingering and user motions. 

35. The system as in claim 31, wherein the selection of the 
operating mode provides for selection of the visual advance- 
ment of the display presentation as at least one of scrolling 

of display music notation, page turn, page flip, and page ^ 
push. 

36. The system as in claim 1, wherein the local presen- 
tation is a video display. 

37. A communications system comprising: 

a plurality of individual music workstations, each com- J0 
prising a music display apparatus for providing a local 
visual display presentation of a selected musical com- 
position and a music input for selectively providing a 
performance data output, responsive to a performance 
by a user of that respective individual music subsystem; }5 

wherein the system further comprises means, responsive 
to the performance data output from each of the plu- 
rality of individual music workstations, to provide a 
combined output of composite virtual performance 
data. 40 

38. The communications system as in claim 37, wherein 
the plurality of individual music subsystems each provide 
means for independently capturing the musical performance 
of the respective user and generate respective individual 
performance data; 45 

wherein the means responsive to the performance data is 
further comprised of means for synchronizing and 
combining the individual performance data from the 
plurality of individual music subsystems to generate 
combined performance data; and 50 

means for communicating said combined performance 
data to the plurality of individual music subsystems; 

wherein at least one of the individual music subsystems 
provide a local presentation representative of the com- 
bined musical performances for all of the communi- 55 
eating plurality of individual music subsystems respon- 
sive to the combined performance data. 

39. The system as in claim 38 wherein the plurality of 
individual music subsystems provide for synchronized dis- 
play presentation of the musical composition. 60 

40. The system as in claim 38, wherein each of the 
individual music subsystems provide for output of indi- 
vidual performance data representative of the musical per- 
formance by the user corresponding to the display presen- 
tation. 65 

41. The system as in claim 38, wherein the selected 
musical composition is performed in discrete time segments, 



wherein the means responsive to the performance data 
generates a synchronization signal for start of performance, 
and each of the time segments is synchronized relative to the 
synchronization signal from the master means responsive to 
the performance data. 

42. The system as in claim 37, wherein the composite 
virtual performance data is communicated back to the indi- 
vidual music subsystems. 

43. The system as in claim 37 wherein the individual 
workstations provide at least one of an audio output and a 
visual presentation, responsive to the composite virtual 
performance data. 

44. ITie system as in claim 37 wherein each of the 
individual workstations is further comprised of a network 
interface subsystem and a music data processor. 

45. The system as in claim 37, wherein the processor is 
responsive to the composite virtual performance data to 
generate a presentation on the local display. 

46. A method for communication with a source of distri- 
bution of musical data representative of a musical score in 
a computer-usable form to a music workstation for video 
presentation of musical notation for the respective musical 
score, the method comprising: 

accepting at least a part of the musical data from the 
source; 

evaluating the at least a part of the musical data to 

determine if the musical data is to be downloaded to the 

respective music workstation; 
selectively storing in local storage the musical data 

responsive to the evaluating; and 
locally displaying a video presentation of the musical 

score responsive to the selectively stored musical data. 

47. The method as in claim 46, further comprised of: 
providing a plurality of separate ones of the music 

workstations, each of which provides for selectively 
storing and locally displaying. 

48. A method for music via performance, for integrating 
simultaneous musical performances from a plurality of 
locations of music display workstations into a cohesive 
whole, the method comprising: 

accepting performance data from each of the plurality of 

music display workstations; 
processing the live performance data into discrete time 

samples; 

communicating the discrete time samples for combination 
into combined virtual performance data; 

communicating the combined virtual performance data to 
at least one of the plurality of music display worksta- 
tions; and 

providing at least one of an audio and a video presentation 
responsive to the combined virtual performance data. 

49. A method of providing a video display presentation of 
a selected musical composition and of a user's performance, 
said method comprising: 

storing composition data representative of the selected 
musical composition; 

generating a video display output responsive to the com- 
position data for the selected musical composition; 

displaying the video display presentation responsive to 
the video display output; 

storing user performance data concurrent with the corre- 
sponding display presentation responsive to the perfor- 
mance by the user; 

comparing the user performance data to the respective 
associated composition data; and 
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modifying the display presentation to reflect the result of 
the comparing concurrent with the performance of the 
composition data. 

50. A music performance system for use by a user in 
providing a performance of a display presentation of a s 
selected musical composition, said system comprising: 

an individual music workstation (IMW) comprising 
a communication interface providing for communications 

of musical composition data corresponding to the 

selected musical composition; 
a computing subsystem providing processing and 

memory for locally storing the musical composition 

data responsive to the communication interface; and 
a music display apparatus for providing a visual display 15 

presentation of the selected musical composition 

responsive to the computing subsystem and the musical 

composition data. 

51. The system as in claim 50, further comprising: 
association means for associating an Instrument Type to 20 

the IMW; 

control means for broadcasting musical display data for 
multiple separate graphical display presentations cor- 
responding to multiple separate respective multiple 
Instrument Types; 25 

wherein the IMW is further comprised of discrimination 
means for discriminating between the multiple separate 
graphical presentations to select a specific one repre- 
sentative of the corresponding respective one of the 
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Instrument Types, responsive to the association means 
and the discrimination means. 

52. The system as in claim 51, further comprising: 

a source of secondary video data representative of a 

secondary video image; 
video controller means for displaying the secondary video 

image as a picture-in-picture within a subpart of the 

visual display presentation. 

53. A communications system comprising: 

an individual music subsystem comprising a music input 
for selectively providing a performance data output, 
responsive to a performance by a user of that respective 
individual music subsystem, and a music data receiver 
for coupling communicated music data for storage in 
memory of the respective individual music subsystem; 
and 

a display for providing a visual presentation to the user 
from at least one of the stored music data and the 
performance data output from the individual music 
subsystem. 

54. The system as in claim 53, further comprising a 
plurality of individual music subsystems, the system further 
comprising: 

means to provide a combined output of composite virtual 
performance data responsive to communicated perfor- 
mance data out from at least two from the plurality of 
the individual music subsystems. 



